atproto-pds-ansible/README.md

# atproto-pds-ansible

Ansible playbook for deploying a self-hosted [Bluesky PDS](https://github.com/bluesky-social/pds) with Caddy (TLS) and Watchtower (auto-updates).

## What it deploys

- **PDS** — Bluesky Personal Data Server (`ghcr.io/bluesky-social/pds:0.4`)
- **Caddy** — Reverse proxy with automatic TLS via Let's Encrypt
- **Watchtower** — Pulls new PDS image versions hourly and recreates containers
- **PDS Backup Service** — rclone + systemd timer for periodic PDS backups to S3, plus Watchtower pre-update hook
- **Monitoring** *(optional)* — Prometheus + Grafana + node_exporter stack with OpenTelemetry metrics and a pre-built PDS Overview dashboard

## Requirements

- A server running Ubuntu 20.04/22.04/24.04 or Debian 11/12/13
- SSH access as the `sysadmin` user (with sudo privileges)
- A domain pointed at the server (A record for the hostname + wildcard `*.` subdomain for handles)
- Ports 80 and 443 open to the internet
- Ansible installed locally

## Setup

```bash
cp inventory.yml.example inventory.yml       # set your server IP
cp vars/main.yml.example vars/main.yml       # set hostname, admin email, S3/SMTP creds
```

Edit `inventory.yml`:

```yaml
all:
  hosts:
    pds:
      ansible_host: 203.0.113.10
```

Edit `vars/main.yml`:

```yaml
pds_hostname: bsky.example.com
pds_admin_email: you@example.com
pds_blobstore_s3_access_key: your-access-key
pds_blobstore_s3_secret_key: your-secret-key
```

## Deploy

```bash
ansible-playbook playbook.yml
```

## What it does

**First run:** the official [PDS installer](https://github.com/bluesky-social/pds/blob/main/installer.sh) bootstraps Docker, generates secrets, and creates the initial database. Ansible then reads the generated secrets from `pds.env`, templates `pds.env`, `compose.yaml`, and `Caddyfile`, and runs `docker compose up -d`.

**Subsequent runs:** the installer is skipped (guarded by `account.sqlite`). Ansible reads the existing secrets from `pds.env` and re-templates all config files. Changes only trigger a restart if content actually changed.

## Variables

### PDS

| Variable | Required | Default | Description |
|---|---|---|---|
| `pds_hostname` | yes | — | Public DNS name for the PDS |
| `pds_admin_email` | yes | — | Admin contact email (used by Caddy for Let's Encrypt) |
| `pds_data_dir` | no | `/pds` | Data directory (must be `/pds` — the installer enforces this) |
| `pds_image_tag` | no | `0.4` | PDS Docker image tag (floating — tracks latest 0.4.x distro release) |
| `pds_blob_upload_limit` | no | `314572800` | Max blob upload size in bytes (300MB) |
| `pds_rate_limits_enabled` | no | `true` | Enable rate limiting |
| `pds_invite_required` | no | `true` | Require invite codes to create accounts |
| `pds_blobstore_type` | no | `disk` | Blob storage backend (`disk` or `s3`) |
| `pds_blobstore_s3_bucket` | yes (if S3) | — | S3 bucket for blob storage |
| `pds_blobstore_s3_region` | yes (if S3) | — | S3 region for blob storage |
| `pds_blobstore_s3_endpoint` | yes (if S3) | — | S3 endpoint for blob storage |
| `pds_blobstore_s3_force_path_style` | no | — | Use path-style S3 addressing |
| `pds_blobstore_s3_access_key` | yes (if S3) | — | S3 access key for blob storage |
| `pds_blobstore_s3_secret_key` | yes (if S3) | — | S3 secret key for blob storage |
| `pds_email_smtp_url` | no | — | SMTP URL (with credentials) for email verification |
| `pds_email_from_address` | no | — | From address for outgoing emails |

### Branding

All optional. Defaults to hostname + "PDS" with neutral colors. See the [PDS customization blog post](https://atproto.com/blog/pds-customization-and-metrics) for details.

| Variable | Default | Description |
|---|---|---|
| `pds_service_name` | hostname + " PDS" | Display name shown on auth screens and emails |
| `pds_logo_url` | — | Logo URL (SVG recommended) |
| `pds_primary_color` | — | Primary color for buttons, links, accents (hex, e.g. `#7c3aed`) |
| `pds_error_color` | — | Error state color |
| `pds_warning_color` | — | Warning state color |
| `pds_info_color` | — | Info state color |
| `pds_success_color` | — | Success state color |
| `pds_home_url` | — | Home page link in footer |
| `pds_terms_of_service_url` | — | Terms of service link in footer |
| `pds_privacy_policy_url` | — | Privacy policy link in footer |
| `pds_support_url` | — | Support link in footer |
| `pds_background_light_url` | — | Background image for auth screens (light mode) |
| `pds_background_dark_url` | — | Background image for auth screens (dark mode) |

### Telemetry & monitoring

Requires a recent PDS image (telemetry added in `@atproto/pds` 0.5.26, distro tag `0.4.5026+`). The `0.4` floating tag already includes this. When enabled, deploys Prometheus + Grafana + node_exporter alongside the PDS. All services bind to `127.0.0.1` — access via SSH tunnel.

| Variable | Default | Description |
|---|---|---|
| `pds_telemetry_enabled` | `false` | Enable OpenTelemetry + monitoring stack |
| `pds_grafana_image_tag` | `13.1.1` | Grafana Docker image tag (pinned — no floating major tag) |

### Backup service

| Variable | Required | Default | Description |
|---|---|---|---|
| `pds_backup_period` | no | `24h` | Periodic backup interval (systemd timer) |
| `pds_backup_s3_bucket` | no | `atprotopds-backup` | S3 bucket for backups |
| `pds_backup_s3_region` | no | `bhs` | S3 region for backup bucket |
| `pds_backup_s3_endpoint` | no | `https://s3.bhs.io.cloud.ovh.net` | S3 endpoint for backup bucket |
| `pds_backup_s3_access_key` | no | — | S3 access key for backups (defaults to blobstore key) |
| `pds_backup_s3_secret_key` | no | — | S3 secret key for backups (defaults to blobstore key) |

## Auto-updates

Watchtower checks every hour (`WATCHTOWER_POLL_INTERVAL=3600`) for new `0.4.x` images on `ghcr.io/bluesky-social/pds:0.4`. When a new image is found, it runs the pre-update backup hook (tarballs `/pds` excluding blocks, uploads to S3), then pulls the new image and recreates the container. Old images are cleaned up automatically.

## Backups

Three backup types, all uploaded to `s3://atprotopds-backup`:

- **Pre-update** — Watchtower lifecycle hook fires before pulling a new PDS image. Runs inside the PDS container using a mounted rclone binary.
- **Pre-deploy** — Ansible task runs before applying config changes. Runs on the host.
- **Periodic** — systemd timer (`pds-backup.timer`), default every 24h. Runs on the host.

Backups exclude `/pds/blocks` (blobs are already in S3). Each backup is a tarball named `{type}-YYYYMMDD-HHMMSS.tar.gz`.

## Monitoring

Set `pds_telemetry_enabled: true` in `vars/main.yml` to enable OpenTelemetry metrics and deploy the monitoring stack (Prometheus + Grafana + node_exporter). The PDS pushes metrics over OTLP to Prometheus, which receives them directly (no collector needed).

**Access Grafana via SSH tunnel:**

```bash
ssh -L 3001:localhost:3001 sysadmin@your-server
```

Then open `http://localhost:3001` and log in with `admin`/`admin` (change on first login). The PDS Overview dashboard is provisioned automatically in a folder named "PDS".

**Ports (all bound to 127.0.0.1):**

| Service | Port | Purpose |
|---|---|---|
| Grafana | 3001 | Dashboard UI |
| Prometheus | 9090 | Metrics query API |
| node_exporter | 9100 | Host metrics |

Prometheus retains 15 days of data by default. Data is stored in Docker volumes, not under `/pds`, so it's not included in PDS backups.

## Post-deploy

Check that the PDS is healthy:

```bash
curl https://your-hostname/xrpc/_health
```

Create an account:

```bash
pdsadmin account create
```

## Teardown

Run on the server to remove everything, use with caution:

```bash
sudo docker compose -f /pds/compose.yaml down
sudo rm -rf /pds
sudo systemctl stop pds 2>/dev/null
sudo systemctl disable pds 2>/dev/null
sudo rm -f /etc/systemd/system/pds.service /usr/local/bin/pdsadmin
sudo systemctl daemon-reload
```

## Notes

- `pds.env` is Ansible-managed. Secrets (JWT, admin password, PLC rotation key) are generated by the installer on first run, then read and preserved by Ansible on subsequent runs. To change config like S3 or SMTP, update `vars/main.yml` and re-run the playbook.
- The installer creates a `pds.service` systemd unit. It's left in place — it just runs `docker compose up -d` on boot. Docker's `restart: unless-stopped` policy also handles this independently.

## CI/CD (Gitea Actions)

Auto-deploys on push to `main`. Set up these in Gitea repo settings (Settings > Actions):

**Variables** (non-sensitive config):

| Variable | Description |
|---|---|
| `SERVER_IP` | Target server IP address |
| `PDS_HOSTNAME` | Public DNS name for the PDS (e.g., `pds.example.com`) |
| `PDS_ADMIN_EMAIL` | Admin contact email |
| `PDS_TELEMETRY_ENABLED` | *(optional)* Set to `true` to enable telemetry + monitoring stack |
| `PDS_SERVICE_NAME` | *(optional)* PDS display name for branding |
| `PDS_PRIMARY_COLOR` | *(optional)* Primary color (hex) for branding |

Outgoing emails are sent from `techops@cosocial.ca` (fixed in the workflow).

**Secrets** (sensitive credentials):

| Secret | Description |
|---|---|
| `SSH_PRIVATE_KEY` | Private SSH key for the `sysadmin` user on the target server |
| `OVH_S3_ACCESS_KEY` | OVH S3 access key for blob storage |
| `OVH_S3_SECRET_KEY` | OVH S3 secret key for blob storage |
| `SMTP_URL` | *(optional)* SMTP URL with credentials (e.g., `smtps://user:pass@smtp.example.com/`) |

The workflow installs Ansible, writes the SSH key from a secret, generates `inventory.yml` and `vars/main.yml` from vars and secrets, and runs the playbook.