Deployment guide

Stack: Django · MySQL · Nginx · Docker Swarm · Cloudflare Tunnel · GitHub Actions

This guide covers the complete setup of a production server from a fresh Ubuntu/Debian droplet to a fully automated deployment pipeline. Follow each section in order on a first-time setup.

The bootstrap is fully scripted. setup.py provisions the VM, creates the Cloudflare Tunnel, ingress rules, DNS records, the Access application protecting SSH, and attaches a Service Auth policy that lets the GitHub Actions deploy pipeline authenticate. Credentials are supplied as flags; the script wires them up.

Todo

• mkdir -p /opt/bucket/storage
• mkdir -p /opt/bucket/media
• install rclone and config — now handled by setup.py (see Section 4)

Table of Contents

1. Prerequisites
2. One-time Cloudflare credentials
3. Server Provisioning
4. Running the Setup Script
5. Directory Structure
6. Manual Cloudflare Steps Still Required
   • 6.1 Verify the tunnel is healthy
7. Post-install: SSH cutover and landing page
   • 7.1 Test SSH through the tunnel
   • 7.2 Close port 22
   • 7.3 Customise the landing page
8. GitHub Actions Self-Hosted Runner
9. GitHub Repository Secrets (via secrets.py)
10. Triggering a Deployment
11. Deployment Methods Explained
12. Monitoring & Logs
13. Rollback Procedure
14. Emergency Server Access
15. Firewall Reference
16. Re-running setup.py
17. Troubleshooting

1. Prerequisites

Accounts and services required

Local machine requirements

• ssh and ssh-keygen available in the terminal
• git installed
• A domain pointed to Cloudflare nameservers
• A password manager — for storing the credentials gathered in Section 2

2. One-time Cloudflare credentials

These are gathered once, stored in a password manager, and reused for every setup.py run. They are never committed to a repo or pushed to GitHub.

2.1 API Token

Cloudflare → My Profile → API Tokens → Create Token → Custom token.

Required permissions:

Set zone resources to All zones (or list each zone explicitly). Copy the token at the end of the creation flow — it is shown only once. This is --cf-api-token.

2.2 Account ID

Cloudflare dashboard → any zone → right sidebar → Account ID. Permanent; doesn’t change. This is --cf-account-id.

2.3 Service Token

Zero Trust → Access → Service Auth → Service Tokens → Create Service Token.

• Name it something memorable, e.g. deploy-pipeline
• Set an expiry (1 year recommended; set a calendar reminder)
• Copy both the Client ID (ends in .access) and Client Secret — the secret is shown only once

These become --cf-service-token-id and --cf-service-token-secret for setup.py, and the matching CF_SERVICE_TOKEN_ID / CF_SERVICE_TOKEN_SECRET in each app’s .env. Same token in both places. It identifies the deploy pipeline as a whole — bootstrap and deploy are the same actor.

Why is the service token created by hand and not by setup.py?

Cloudflare returns the secret exactly once, at creation. If setup.py generated it, the secret would have to be captured to a file or printed to stdout, and lost values would force a rotation that invalidates every app’s GitHub secret simultaneously. By creating the token once in the dashboard and storing it in a password manager, the same value can be reused across VM rebuilds, multiple setup.py runs, and every app repo’s deploy pipeline — with no irrecoverable state on the VM.

2.4 R2 API Token (for rclone)

setup.py installs rclone unconditionally and writes a pre-configured R2 remote at ~/.config/rclone/rclone.conf whenever R2 credentials are passed. This is what powers ad-hoc bucket copies, restore drills, and the project’s backup tooling.

Cloudflare dashboard → R2 → Manage R2 API Tokens → Create API Token:

• Permissions: Object Read & Write (or read-only if this VM should never push to buckets)
• Specify bucket(s): scope to the buckets the server actually needs, not “Apply to all buckets”
• TTL: open-ended is fine; rotate when team changes

Cloudflare returns three values:

Both the Access Key ID and Secret Access Key are shown only once. Store them in the password manager alongside the other bootstrap credentials.

The endpoint URL is https://<ACCOUNT_ID>.r2.cloudflarestorage.com. setup.py builds it from --cf-account-id automatically; pass --r2-account-id only to use a different account for R2 than for the tunnel (rare).

Store these safely

Suggested password-manager entry:

Production server bootstrap
  CF_API_TOKEN              cfut_...
  CF_ACCOUNT_ID             f47...
  CF_SERVICE_TOKEN_ID       xxxx.access
  CF_SERVICE_TOKEN_SECRET   yyyy
  R2_ACCESS_KEY_ID          aaaa...
  R2_SECRET_ACCESS_KEY      bbbb...

3. Server Provisioning

Create the droplet

Provision a fresh Ubuntu 22.04 LTS or Debian 12 server. Minimum recommended specs:

Generate an SSH key pair

On the local machine:

ssh-keygen -t ed25519 -C "prod-server-deploy" -f ~/.ssh/prod_server

This produces two files:

• ~/.ssh/prod_server — private key (never shared, added to GitHub secrets later via secrets.py)
• ~/.ssh/prod_server.pub — public key (added to the server)

Add the public key to the server

When provisioning through DigitalOcean, paste the contents of ~/.ssh/prod_server.pub into the SSH key field. Alternatively, after the server is running:

ssh-copy-id -i ~/.ssh/prod_server.pub root@<server-ip>

Verify SSH access

ssh -i ~/.ssh/prod_server root@<server-ip>

4. Running the Setup Script

server/setup.py handles most of the provisioning: installs Docker, configures UFW, creates application directories, brings up the Cloudflare Tunnel and the landing-page nginx, creates DNS records, and configures the Access application that gates SSH.

There are two ways to run it: from GitHub Actions (preferred — secrets come from .env synced to GitHub Secrets, no hand-typing), or by hand on the server (the traditional way; same script, same flags).

Option A — Dispatch from GitHub Actions (preferred)

1. Confirm .env at the repo root has the required values (see .env.example).
2. Sync them to GitHub Secrets: python3 script/secrets.py --push.
3. In the GitHub UI: Actions → server-setup → Run workflow. Fill in the form (target host, optional force/dry-run).
4. The workflow ships server/setup.py to the target via SSH and runs it with all the secrets as flags. Output appears in the workflow log.

This route suits provisioning a new server: every credential comes from the same source of truth (.env), with nothing to mistype.

Option B — Run by hand on the server

Transfer the script and run it directly. Useful for one-off debugging or when GitHub Actions isn’t available.

scp -i ~/.ssh/prod_server server/setup.py root@<server-ip>:/root/setup.py

Robotic mode — recommended

Run with all five required credentials. --app-domain is repeatable; supply one per public hostname the apps need. --r2-access-key-id and --r2-secret-access-key are optional — supply them to have setup.py write a pre-configured rclone remote for R2.

sudo python3 setup.py \
  --cf-api-token             "cfut_..." \
  --cf-account-id            "f47..." \
  --cf-service-token-id      "xxxx.access" \
  --cf-service-token-secret  "yyyy" \
  --domain                   "example.com" \
  --tunnel-name              "prod-server" \
  --app-domain               "myordbok.com:http://localhost:8000" \
  --app-domain               "zaideih.com:http://localhost:8080" \
  --r2-access-key-id         "aaaa..." \
  --r2-secret-access-key     "bbbb..."

rclone itself is installed even when the R2 flags are omitted — the binary is useful for ad-hoc work regardless. Only the [r2] remote in ~/.config/rclone/rclone.conf is gated on the credentials being present.

Manual-tunnel mode (legacy)

If the tunnel already exists in the Cloudflare dashboard and only the server-side install is needed (Docker, tunnel container, landing nginx), pass the tunnel token directly:

sudo python3 setup.py --cloudflare-token <TUNNEL_TOKEN>

This skips all robotic steps; public hostnames, the Access application, the Access policy, and the DNS records must then be added by hand. Robotic mode is preferred — manual mode exists for migration cases where the tunnel and DNS already exist.

Other options

Adding or renaming admin subdomains

Every admin subdomain (DNS record + tunnel ingress + optional Access app) is declared in a single dict named admin_subdomains near the top of main() in setup.py. Edit one place and DNS, ingress, Access wiring, and the post-install summary all follow.

admin_subdomains = {
    "ssh": {
        "service": "tcp://localhost:22",
        "protect_with_access_app": True,
        "app_name": f"SSH — {admin_domain}",
    },
    # Browser-rendered SSH terminal — DNS + ingress only (see below).
    "terminal": {
        "service": "ssh://localhost:22",
        "ca_cert": True,   # app holding the short-lived certificate CA
    },
    # Add new internal proxies here, e.g.:
    # "grafana":   {"service": "http://localhost:3000"},
    # "portainer": {"service": "https://localhost:9443"},
}

To rename ssh to something else (e.g. access), just change the dict key — setup.py will create a DNS record, ingress rule, Access app, and Service Auth policy on the new hostname. The old Cloudflare resources at the previous hostname stay in place and need manual cleanup in the dashboard.

SSH uses the tcp:// scheme, not ssh://. Remotely managed tunnels — configured through the API, as this script does — route SSH as a raw TCP service; the ssh:// scheme belongs to the older local-config-file model and is silently ignored. The operator-side connection still uses cloudflared access tcp (see Section 7).

The terminal entry is different. It uses ssh:// deliberately (the browser-render flow expects it) and omits protect_with_access_app, so setup.py creates only its DNS record and ingress rule — no Access app, policy, or service-token wiring. The rest of the browser-terminal setup (the self-hosted app and its email policy) is configured once in the dashboard. The short-lived-certificate CA it relies on, however, can be trusted on the server automatically: pass --add-ca-public-key (see below). The CA itself is still generated by hand in the dashboard — setup.py only resolves the Access app flagged ca_cert in admin_subdomains (the terminal app), fetches that app’s CA, and trusts an existing one. It never creates the CA.

The one-time dashboard setup for terminal.<admin_domain>:

Two requirements that are easy to miss. The operator’s login email prefix must match a Unix username on the server (for example, jdoe@example.com logs in as jdoe), or the certificate principal will not match. And the browser terminal needs these sshd key-exchange algorithms; add them if absent: curve25519-sha256, curve25519-sha256@libssh.org, ecdh-sha2-nistp256, ecdh-sha2-nistp384, ecdh-sha2-nistp521.

What the script does

Verify the install completed

docker info
docker compose version
docker service ls
ufw status

Setup logs are written to /var/log/setup.log.

5. Directory Structure

/opt/
├── infra/
│   └── landing/               # Vanilla nginx landing-page stack
│       ├── docker-compose.yml # nginx:alpine, port 80, mounts the two below
│       └── conf/
│           └── nginx.conf     # The server block (try_files, root, etc.)
│
├── cloudflare-tunnel/         # Cloudflare Tunnel as its own stack
│   ├── docker-compose.yml     # cloudflared:latest, network_mode: host
│   └── .env                   # TUNNEL_TOKEN (chmod 600)
│
├── myordbok/                  # Application deployment directory
│   ├── docker.production.yml  # Copied here by deploy.yml on each deploy
│   └── .env                   # Recreated from GitHub secret on each deploy
│
├── zaideih/                   # Second application — same shape
│   ├── docker.production.yml
│   └── .env
│
├── apps/
│   └── html/                  # Landing page document root (mounted ro into landing nginx)
│
├── bucket/                    # Shared persistent storage
│   ├── storage/
│   └── media/
│
├── django/
│   └── media/                 # Django media file uploads
│
└── mysql/
    └── data/                  # MySQL data directory (owned by uid 999)

~/.config/rclone/
└── rclone.conf                # R2 remote, mode 600, owned by SUDO_USER

/opt/<app>/.env is not a manually managed file. It is recreated from the ENV_FILE_CONTENT GitHub secret on every deployment run. The marker that an app has been deployed to this host is the existence of /opt/<app>/.env.

~/.config/rclone/rclone.conf is owned by the user who invoked sudo python3 setup.py (resolved via $SUDO_USER), not root — so rclone lsd r2: works without sudo from the operator’s shell.

6. Manual Cloudflare steps still required

Robotic mode handles SSH-related Access configuration end-to-end. One thing still benefits from a quick dashboard check:

6.1 Verify the tunnel is healthy

Zero Trust → Networks → Tunnels — find the tunnel named in --tunnel-name. Status should read Healthy with one connector. If it reads Inactive, check the cloudflared container on the server:

docker logs --tail 50 cloudflare-tunnel

The most common cause of an inactive tunnel after setup.py finishes is a wrong tunnel token in /opt/cloudflare-tunnel/.env — usually because robotic mode was bypassed and the manual --cloudflare-token value didn’t match the named tunnel.

7. Post-install: SSH cutover and landing page

With setup.py complete, the Cloudflare tunnel is up, ingress rules are routing, and a placeholder landing page is serving on port 80. Two items remain before the server is done.

7.1 Test SSH through the tunnel (before closing port 22)

Install cloudflared on the local machine:

# macOS
brew install cloudflared

# Linux (Debian/Ubuntu)
wget -q https://github.com/cloudflare/cloudflared/releases/download/2025.4.0/cloudflared-linux-amd64.deb
sudo dpkg -i --force-overwrite cloudflared-linux-amd64.deb

Open a local listener that forwards a localhost port through the tunnel:

cloudflared access tcp --hostname ssh.<admin-domain> --url localhost:2222 &

Then SSH to the local forward in another terminal:

ssh -p 2222 -i ~/.ssh/prod_server root@localhost

For interactive (browser) login, Cloudflare opens a tab the first time the listener starts. For service-token-based testing (matching what the deploy pipeline does), pass the token to the listener:

cloudflared access tcp \
    --hostname ssh.<admin-domain> \
    --url localhost:2222 \
    --service-token-id <TOKEN_ID> \
    --service-token-secret <TOKEN_SECRET> &
ssh -p 2222 -i ~/.ssh/prod_server root@localhost

If this connects without prompting for a browser login, the Service Auth policy created by setup.py is working correctly and the deploy pipeline will authenticate the same way. Stop the listener with kill %1 when done.

7.2 Close port 22

Only close port 22 after the tunnel-based SSH test above succeeds.

From an active SSH session on the server:

sudo ufw delete allow ssh
sudo ufw status

Port 22 is now closed. Future terminal access goes through the tunnel. The DigitalOcean web console remains as an emergency fallback (see Section 14).

7.3 Customise the landing page

Anything not matched by a specific tunnel ingress rule falls through the catch-all to localhost:80, where the landing-page nginx serves whatever is in /opt/apps/html/. This is the lightweight nginx:alpine container at /opt/infra/landing/. There is no admin UI, no DB, no template injection points. The container does one thing: serve /opt/apps/html as a static site.

The repo’s apps/html/ is the source for that page. Edit files there, commit, then dispatch the apps-deploy workflow from GitHub Actions — it builds (if apps/html/package.json exists) and rsyncs to /opt/apps/html on the server. That’s the preferred update path: edit-in-repo → workflow → done.

For one-off changes without going through the repo:

# Single file
sudo cp my-landing.html /opt/apps/html/index.html

# Or a whole SPA / static-site build output
sudo rsync -av ./dist/ /opt/apps/html/

But anything that doesn’t come from the repo will be overwritten by the next apps-deploy run, so use sparingly.

No restart, no reload — nginx reads files from the bind mount on every request. The server block uses try_files $uri $uri/ /index.html =404;, which means:

• Concrete files (style.css, script.js, favicon.ico, assets/logo.png, etc.) resolve via $uri and serve as static files directly from disk.
• Requests for client-side-routed SPA paths (e.g. /about, /users/42) don’t match a file or directory, so they fall back to /index.html — the SPA shell — which then handles routing in the browser.
• Genuinely missing paths with no SPA shell to fall back to return 404, not an infinite loop.

To change the nginx server block itself (custom headers, gzip, additional locations), edit /opt/infra/landing/conf/nginx.conf on the host and recreate the container:

cd /opt/infra/landing && sudo docker compose up -d --force-recreate landing

What hits this landing page? Three things:

• http://<server-ip>/ — direct HTTP to the server’s public IP
• https://<server-ip>/ — direct HTTPS won’t reach here (no TLS on the host); Cloudflare’s edge handles HTTPS for domains pointed at this server
• Any unknown-hostname request that reaches the tunnel — Cloudflare routes via the catch-all ingress rule to localhost:80

Per-app domains (myordbok.example.com, zaideih.example.com) and admin domains (ssh.<domain>) bypass the landing page entirely — they match specific tunnel ingress rules that route to their app/SSH ports directly.

Why a vanilla nginx container? The landing layer only serves static files — TLS terminates at the Cloudflare edge and host-to-app routing happens at the tunnel via localhost:<port>. So the container does one thing: serve /opt/apps/html as a static site with an SPA try_files fallback. The config is deliberately minimal — one server block, one root, one try_files.

8. GitHub Actions Self-Hosted Runner

The deploy_local_vm job in deploy.yml runs directly on the production server using a self-hosted runner. This runner is not installed by setup.py and must be registered manually.

Register the runner

1. GitHub repository → Settings → Actions → Runners
2. Click New self-hosted runner → Linux / x64
3. Run the displayed commands on the production server:

mkdir actions-runner && cd actions-runner
curl -o actions-runner-linux-x64-2.x.x.tar.gz -L \
    https://github.com/actions/runner/releases/download/...
tar xzf ./actions-runner-linux-x64-2.x.x.tar.gz
./config.sh --url https://github.com/<org>/<repo> --token <RUNNER_TOKEN>

Install as a system service

sudo ./svc.sh install
sudo ./svc.sh start
sudo ./svc.sh status

The runner should show as Idle at Settings → Actions → Runners in GitHub.

Create the PAT for runner status checks

check_local_vm_runner polls the GitHub API to detect whether the runner is online before attempting a local deploy.

1. GitHub → Settings (account) → Developer settings → Personal access tokens → Fine-grained tokens
2. Repository access: the deployment repository
3. Grant read-only permission for Actions
4. Copy the token. It goes into each app’s .env as VM_RUNNER_STATUS_PAT and secrets.py --push ships it to GitHub.

9. GitHub Repository Secrets (via secrets.py)

Two secrets.py scripts exist. This section is about the per-app one — it lives in each Django app repo (e.g. myordbok, zaideih), reads that app’s own .env, and pushes secrets to that app’s GitHub repo so its deploy pipeline (deploy.yml) can authenticate. There is also lethil/script/secrets.py which does the same kind of thing but for the lethil infra repo’s own .env — driving the server-setup and apps-deploy workflows. Different .env files, different target repos, same idea.

Repository secrets are managed by secrets.py, which lives alongside (or inside) each app repo. The source of truth is each app’s .env. See secrets.md for the full reference.

How values flow

Password manager        Each app's .env (Zone 3)        GitHub Actions Secrets
────────────────        ────────────────────────        ──────────────────────
SERVER_HOSTNAME    ──→  SERVER_HOSTNAME           ──→  SERVER_HOSTNAME
                        SSH_PRIVATE_KEY_PATH      ──→  SSH_PRIVATE_KEY  (file contents)
                        VM_RUNNER_STATUS_PAT      ──→  VM_RUNNER_STATUS_PAT
CF_SERVICE_TOKEN_ID──→  CF_SERVICE_TOKEN_ID       ──→  CF_SERVICE_TOKEN_ID
CF_SERVICE_TOKEN_SECRET→ CF_SERVICE_TOKEN_SECRET  ──→  CF_SERVICE_TOKEN_SECRET
                        (Zone 1 lines bundled)    ──→  ENV_FILE_CONTENT

CF_SERVICE_TOKEN_ID / CF_SERVICE_TOKEN_SECRET are the same values passed to setup.py. One credential, three locations: password manager (durable), .env (per-app local working copy), GitHub Secret (consumed by the runner).

Push secrets

In each app repo:

secrets.py --check                # confirm gh CLI auth + .env parses
secrets.py --status               # see what would change
secrets.py --push --dry-run       # preview
secrets.py --push                 # ship to GitHub

Required secrets

GitHub environments

Tunnel and SSH deploy jobs are scoped to named environments. Create these once per repo:

Repository → Settings → Environments → New environment

Environments support deployment gates (required reviewers, wait timers, environment-scoped secrets).

Verify secrets are set

secrets.py --list

Prints all secret names currently on the repo. Values are never shown.

10. Triggering a Deployment

Deployments are triggered by pushing to the master branch with a commit message starting with deploy:.

Commit message format

deploy: <description> [optional-tag]

Deployment method selection

[tunnel] and [ssh] are mutually exclusive. If both appear in the same commit, [tunnel] takes priority and [ssh] is ignored.

Of these, only [ssh] is implemented today. The untagged self-hosted path and [tunnel] are planned — see Section 11. The selection logic above describes the intended end state.

Example

git add .
git commit -m "deploy: update homepage layout [ssh]"
git push origin master

Watch the pipeline

Repository → Actions tab → most recent run.

11. Deployment Methods Explained

The deploy methods below are driven by the application repository’s workflow, not by lethil. lethil provisions the server; the app repo deploys to it. Of the three, only native SSH is implemented today — self-hosted and tunnel are planned.

Method 1 — Local VM (self-hosted runner) — planned

Status: not yet implemented. Runner registration and the self-hosted deploy path live in the application repository.

Intended behaviour: a self-hosted runner on the target executes the deploy locally, with no SSH or network tunnelling involved. Intended as a free, real target for testing production-ready apps.

Method 2 — Cloudflare Tunnel ([tunnel]) — planned

Status: not yet implemented. Deferred as a later hardening step once the SSH path is proven in regular use.

Intended behaviour: the GitHub-hosted runner opens a cloudflared access tcp listener authenticated by the Service Token configured by setup.py, SSHes to the forwarded local port, copies the compose file and .env, and runs the deploy. Port 22 stays closed. The server side is already wired up by setup.py (Access app + Service Auth policy on ssh.<domain>); only the workflow side is outstanding.

Best for: remote servers behind closed firewalls.

Method 3 — Native SSH ([ssh])

When: [ssh] in the commit message and [tunnel] not present.

How: uses appleboy/scp-action and appleboy/ssh-action over standard SSH. Requires port 22 reachable from GitHub Actions IP ranges.

Best for: environments without Cloudflare Tunnel, or initial setup before the tunnel is configured.

Docker image tagging

Every build pushes two tags per image:

ghcr.io/<repo>/django-app:latest
ghcr.io/<repo>/django-app:<commit-sha>

:latest is what the running stack uses. The SHA tag stays in the registry for rollbacks.

12. Monitoring & Logs

GitHub Actions deployment summary

After each successful deploy, a summary is written to the job page: Actions → <run> → <job> → Summary.

Docker service status

docker service ls
docker service ls --filter name=myordbok_web
docker service ps myordbok_web

Application logs

docker service logs myordbok_web --follow
docker service logs myordbok_nginx --follow
docker service logs myordbok_db --follow
docker service logs myordbok_web --tail 100

Cloudflare Tunnel logs

docker logs -f cloudflare-tunnel
docker exec cloudflare-tunnel cloudflared tunnel info

System resource usage

free -h
df -h
docker system df

13. Rollback Procedure

Every build tags images with the commit SHA. To roll back:

1. Find the target SHA — from the Actions deploy summary, or git log --oneline.
2. Pull the tagged image:

   docker pull ghcr.io/<repo>/django-app:<target-sha>
   

3. Pin the tag: edit /opt/<app>/docker.production.yml to replace :latest with :<target-sha> for the django-app service, then redeploy:

   cd /opt/<app>
   docker stack deploy -c docker.production.yml --with-registry-auth --detach=false <app>
   

4. Verify:

   docker service ls --filter name=<app>_web
   docker service logs <app>_web --tail 50
   

14. Emergency Server Access

Via Cloudflare Tunnel SSH (primary remote method)

Open a listener forwarding a local port through the tunnel (see Section 7.1), then SSH to it:

cloudflared access tcp --hostname ssh.<admin-domain> --url localhost:2222 &
ssh -p 2222 -i ~/.ssh/prod_server root@localhost

For non-interactive access (no browser login), add the service token to the listener:

cloudflared access tcp \
    --hostname ssh.<admin-domain> \
    --url localhost:2222 \
    --service-token-id <TOKEN_ID> \
    --service-token-secret <TOKEN_SECRET> &
ssh -p 2222 -i ~/.ssh/prod_server root@localhost

Via DigitalOcean Web Console (no network required)

Connects directly through the hypervisor — bypasses UFW, closed ports, and broken networking.

1. Droplets → <name> → Access
2. Launch Droplet Console

Re-opening port 22 if needed

sudo ufw allow ssh
sudo ufw status

15. Firewall Reference

UFW is configured by setup.py. Rules in effect after setup:

sudo ufw status verbose
sudo ufw allow <port>/tcp
sudo ufw delete allow <port>/tcp
sudo ufw reload

16. Re-running setup.py

setup.py is a one-shot bootstrap, but most steps are idempotent — apt installs, swap, UFW rules, directory creation, Docker install, swarm init, compose stacks. Running it again is safe for picking up incremental fixes.

The Cloudflare-side operations are partly idempotent:

The service token is durable. Its credentials are inputs to setup.py, stored in the password manager and reused across runs. There is no secret on the VM that gets lost on reinstall. Tearing down the VM and provisioning a fresh one with the same flags reattaches the same service token to a fresh Access policy on a fresh tunnel — the apps’ GitHub secrets continue to work without rotation.

Rotating the service token (when required)

Service tokens expire, and proactive rotation after team changes is sometimes warranted. The flow:

1. Cloudflare dashboard → Zero Trust → Access → Service Auth → Service Tokens → Create Service Token (or Refresh an existing one — same effect, secret returned once)
2. Update the password manager with the new ID and secret
3. Update each app’s .env (Zone 3): new CF_SERVICE_TOKEN_ID and CF_SERVICE_TOKEN_SECRET
4. In each app repo: secrets.py --push (use --only CF_SERVICE_TOKEN to scope it)
5. Re-run setup.py with the new credentials so the Access policy is re-pointed at the new token
6. Delete the old token in the dashboard

Order matters: keep the old token until step 5 succeeds, or in-flight deploys will fail.

17. Troubleshooting

Build job does not start

The commit message must start with deploy: (lowercase, colon, space). The build job has this condition:

if: startsWith(github.event.head_commit.message, 'deploy:')

A message like Deploy: ... or deploy ... (no colon) will not trigger.

check_local_vm_runner reports runner as offline

Check 1: verify the runner service:

cd ~/actions-runner && sudo ./svc.sh status
sudo ./svc.sh start    # if stopped

Check 2: verify VM_RUNNER_STATUS_PAT is set and not expired. A missing or expired token causes the API call to return an error, treated as offline.

Cloudflare Tunnel deploy fails with websocket: bad handshake

This is the auth layer — Cloudflare Access rejected the service token at the WebSocket handshake. Check, in order:

1. The Access application for ssh.<domain> exists (Zero Trust → Access → Applications). It should — setup.py creates it. If it’s missing, re-run setup.py.

2. The Access app has a Service Auth policy (not “Allow”) referencing the service token. Click the app → Policies tab. The policy’s Action must be Service Auth; Action: Allow with a service token in the include rule will produce this exact error.

3. The token in GitHub Secrets matches the one attached to the policy. CF_SERVICE_TOKEN_ID ends in .access. If they drifted (e.g. the token was rotated in the dashboard but secrets.py --push wasn’t run), redeploys fail until they’re resynced.

4. Tunnel is healthy:

docker logs --tail 50 cloudflare-tunnel

MySQL readiness check times out

The pipeline waits up to 100 seconds for MySQL to accept connections.

docker service logs <app>_db --tail 100

Common causes: wrong MYSQL_ROOT_PASSWORD in ENV_FILE_CONTENT, or insufficient disk space at /opt/mysql/data.

Deployment health check fails (Replicas: 0/2 or 1/2)

docker service logs <app>_web --tail 100

The Dump Service Logs on Failure step in the pipeline captures these automatically and displays them in the Actions run output.

cp fails when installing cloudflared in CI

If the GitHub Actions run shows cp: '/usr/bin/cloudflared' and '/usr/local/bin/cloudflared' are the same file, the runner image already has cloudflared preinstalled as a symlink. Use dpkg -i --force-overwrite and drop any manual cp step — dpkg will manage the binary location itself.

Service token verification fails at the start of setup.py

Service token with client_id ‘xxxx.access’ was not found in account …

setup.py looks up the token in the supplied account before doing anything else. Causes:

• --cf-service-token-id is wrong (typo, or the token name/UUID was copied instead of the Client ID — Client ID always ends in .access)
• --cf-account-id is wrong (wrong account, or token created in a different account)
• The token was deleted in the dashboard

Fix any of those and re-run.

rclone lsd r2: returns NoSuchBucket or InvalidAccessKeyId

The remote is configured but the credentials aren’t matching what Cloudflare expects.

1. Check the config file exists and is owned by the user running rclone (not root):

ls -l ~/.config/rclone/rclone.conf
# Expected: -rw------- 1 <user> <user> ...
cat ~/.config/rclone/rclone.conf

2. Confirm the endpoint matches the account ID. The line should read endpoint = https://<account-id>.r2.cloudflarestorage.com. Account ID is the same one in the Cloudflare dashboard sidebar, not the zone ID.

3. Confirm the R2 API token has access to the bucket being listed. R2 tokens can be scoped per-bucket — Apply to all buckets is permissive but auditable; per-bucket scoping is safer but easier to misconfigure.

4. Verify with a one-shot call:

rclone --config ~/.config/rclone/rclone.conf lsd r2:

To rewrite the config, re-run setup.py --force with the R2 flags. Without --force, the existing [r2] section is preserved.

docker: permission denied while trying to connect to the docker API

permission denied while trying to connect to the docker API at unix:///var/run/docker.sock

setup.py adds the invoking user to the docker group, but Linux only refreshes group memberships when a process starts. The current shell started before setup.py ran, so it still has the old groups.

Two options:

# Activate the group in the current shell only:
newgrp docker

# Or pick it up everywhere by re-logging in:
exit       # then SSH back in

After either of those, docker ps works without sudo. Confirm with:

groups | grep -o docker

If docker appears in the output, the membership is active. If setup.py ran but did not add the account to the group (for example, run directly as root with no SUDO_USER set), add it manually:

sudo usermod -aG docker $USER
newgrp docker

The watch-out: shell scripts and command substitutions like $(docker ps -q -f name=...) run in subshells, which inherit the parent shell’s groups. So sudo docker exec $(docker ps -q ...) runs the outer command as root but the inner one as the normal user — and the inner one fails with the permission error above, leaving the outer docker exec with no container ID to operate on. Fix the group membership instead of sprinkling sudo inside command substitutions.

Landing page doesn’t show, or assets return 502

The landing page should appear when the server is reached over plain HTTP, with or without a hostname header:

curl -sI http://<server-ip>/
curl -sI -H "Host: nothing.invalid" http://<server-ip>/

Both should return 200 OK with Server: nginx/... and a Content-Length matching the /opt/apps/html/index.html file size.

Container running?

docker ps --filter name=landing
# Expected: STATUS shows 'Up <duration>'

If it’s not running, check the logs:

docker logs --tail 50 landing

The most common reason for the container failing to start is a syntax error in /opt/infra/landing/conf/nginx.conf. nginx -t runs at startup and the error will be in the logs verbatim.

Wrong files served (or empty 403 Forbidden)?

# What's on the host?
ls -la /opt/apps/html/

# What does the container see?
docker exec landing ls -la /usr/share/nginx/html/

The two listings should match. If the container is empty or shows different files, the bind mount didn’t apply — docker compose up -d --force-recreate landing from /opt/infra/landing/ will rebuild with the mount.

Port conflict?

If docker logs landing shows bind() to 0.0.0.0:80 failed (98: Address already in use), something else is already listening on port 80. Find what:

sudo lsof -i :80
# or
sudo ss -tlnp | grep ':80'

If nginx-proxy-manager appears in the output, follow the upgrade-cleanup note in Section 5.

SPA deep links return 404?

The server block uses try_files $uri $uri/ /index.html =404;. Deep links work as long as /opt/apps/html/index.html exists. If it doesn’t, deep links 404 because the fallback target doesn’t exist. Drop a real index.html into the directory and reload the browser.