Step 13.3: nightly database backups

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

.dockerignore

@@ -12,5 +12,7 @@
 memory-bank
 docs
 deploy/backups
+deploy/.check-backups
+deploy/.check.env
 # The source map art is only an input to the asset script (decision 0.5.37).
 apps/web/public/assets/map

.gitignore

@@ -9,3 +9,8 @@ coverage/
 # Playwright output
 apps/web/test-results/
 apps/web/playwright-report/
+
+# Local backup output (deploy/compose.yml)
+deploy/backups/
+deploy/.check-backups/
+deploy/.check.env

DECISIONS.md

@@ -1315,3 +1315,13 @@ Implementation decisions not spelled out in `memory-bank/implementation-plan.md`
 - **`APP_BUILD` overrides the git commit in the build info**, because image builds have no `.git`; the README's update command passes the checkout's short hash.
 - **Caching:** only Vite's content-hashed bundles and fonts are marked immutable. The generated map art keeps its plain names, so it isn't; the service worker precache versions it anyway. The shell, service worker, its registration script and the manifest are `no-cache`, so a new deploy is picked up on the next visit (auto-update, Step 11.1).
 - **The 13.2 manual check is scripted locally** (guest sign-up and a tutorial run through Caddy, with the browser's Origin header); loading the real site on desktop and a phone is left to the user, since it needs the home server, DNS and port forwarding.
+
+## Step 13.3
+
+- **Backups run in the stack** as a `backup` service on the same Postgres image (so `pg_dump` matches the server version), with a small shell scheduler rather than cron: one process, logs in `docker compose logs backup`, nothing to install on the host.
+- **Off-server storage is a host folder** (`BACKUP_DIR`), as chosen: a NAS share mounted on the server. The README warns that an unmounted share would put dumps on the server's own disk.
+- **Retention is age-based**: dumps older than `BACKUP_KEEP_DAYS` (default 14) are deleted after each successful backup, so a failing backup never deletes the last good ones by itself running.
+- **Dumps use pg_dump's custom format** (compressed, restorable with `pg_restore --clean --if-exists --no-owner`), written under a `.partial` name and renamed when complete.
+- **`restore.sh` defaults to a separate `deadhead_restore` database**, so the routine test restore never touches live data; recovering the live database is a documented, explicit command with the server stopped.
+- **The step's test compares a player row by row**: after a guest's sign-up and tutorial run, the check backs up, restores into a fresh database and compares that player's rows across nine tables as JSON; they match exactly.
+- **`check.sh` disables Git Bash's path conversion** (`MSYS_NO_PATHCONV`) so container paths reach Docker unchanged on Windows; its temporary settings file sits next to the compose file (`deploy/.check.env`, git- and docker-ignored) for the same reason.

deploy/.env.example

@@ -19,3 +19,10 @@ POSTGRES_PASSWORD=
 BETTER_AUTH_SECRET=
 
 LOG_LEVEL=info
+
+# Nightly database backups: a folder on this machine, e.g. a NAS share mounted
+# here. Dumps older than BACKUP_KEEP_DAYS are deleted.
+BACKUP_DIR=/mnt/nas/deadhead-backups
+BACKUP_HOUR=3
+BACKUP_KEEP_DAYS=14
+TZ=UTC

deploy/README.md

@@ -41,6 +41,37 @@ deploy/check.sh
 ```
 
 This builds and starts the whole stack under a separate project name with
-throwaway secrets, on port 8088. It checks the API, the app, caching and a guest
+throwaway secrets, on port 8088. It checks the API, the app, caching, a guest
-sign-up with a tutorial run, then removes everything. It never touches the real
+sign-up with a tutorial run, and a backup restored into a fresh database. Then
-stack.
+it removes everything. It never touches the real stack.
+
+## Backups
+
+The `backup` service writes a compressed dump of the database every night at
+`BACKUP_HOUR`:00 (`TZ`, default UTC) to `BACKUP_DIR`, and deletes dumps older
+than `BACKUP_KEEP_DAYS`. Point `BACKUP_DIR` at a folder that lives on another
+disk, such as a NAS share mounted on the server (for example through
+`/etc/fstab`). Make sure the share is mounted before Docker starts, or the
+dumps land on the server's own disk.
+
+Take a backup now:
+
+```sh
+docker compose exec backup /scripts/backup.sh
+```
+
+Test a restore without touching the live data. This loads the newest dump into
+a separate `deadhead_restore` database:
+
+```sh
+docker compose exec backup /scripts/restore.sh
+```
+
+Recover the live database from a dump (stop the server first so nothing writes
+while it loads):
+
+```sh
+docker compose stop server
+docker compose exec backup /scripts/restore.sh /backups/deadhead-<stamp>.dump deadhead
+docker compose start server
+```

deploy/backup/backup.sh

@@ -0,0 +1,13 @@
+#!/bin/sh
+# One backup now (Step 13.3): a compressed pg_dump of the game database into
+# /backups, then dumps older than BACKUP_KEEP_DAYS are deleted. Connection
+# settings come from the PG* environment variables.
+set -eu
+
+stamp="$(date -u +%Y%m%dT%H%M%SZ)"
+file="/backups/deadhead-$stamp.dump"
+# Written under a temporary name, so a half-written dump never looks complete.
+pg_dump --format=custom --file="$file.partial"
+mv "$file.partial" "$file"
+find /backups -maxdepth 1 -name 'deadhead-*.dump' -mtime +"${BACKUP_KEEP_DAYS:-14}" -delete
+echo "backup: wrote $file ($(du -h "$file" | cut -f1))"

deploy/backup/restore.sh

@@ -0,0 +1,15 @@
+#!/bin/sh
+# Restores a dump into a database (Step 13.3):
+#   restore.sh [dump] [database]
+# Defaults: the newest dump in /backups, into a new database "deadhead_restore"
+# (created if missing), so the live database is untouched. To replace the live
+# database, stop the server first and pass its name.
+set -eu
+
+dump="${1:-$(ls -1 /backups/deadhead-*.dump | sort | tail -n 1)}"
+database="${2:-deadhead_restore}"
+if ! psql -d postgres -tAc "SELECT 1 FROM pg_database WHERE datname = '$database'" | grep -q 1; then
+  createdb "$database"
+fi
+pg_restore --clean --if-exists --no-owner --dbname="$database" "$dump"
+echo "restore: $dump -> $database"

deploy/backup/schedule.sh

@@ -0,0 +1,13 @@
+#!/bin/sh
+# Runs backup.sh every day at BACKUP_HOUR:00 (in TZ, default UTC).
+set -eu
+
+hour="${BACKUP_HOUR:-3}"
+echo "backup: daily at ${hour}:00 ${TZ:-UTC}, keeping ${BACKUP_KEEP_DAYS:-14} days, into /backups"
+while true; do
+  now="$(date +%s)"
+  next="$(date -d "today ${hour}:00" +%s)"
+  if [ "$next" -le "$now" ]; then next="$(date -d "tomorrow ${hour}:00" +%s)"; fi
+  sleep "$((next - now))"
+  /scripts/backup.sh || echo "backup: FAILED" >&2
+done

deploy/check.sh

@@ -4,10 +4,12 @@
 #   deploy/check.sh            from the repository root or deploy/
 # Exits non-zero if any check fails. Needs Docker with Compose v2.
 set -euo pipefail
+# Git Bash on Windows would rewrite container paths like /scripts/… otherwise.
+export MSYS_NO_PATHCONV=1
 
 cd "$(dirname "$0")"
 project="deadhead-check"
-env_file="$(mktemp)"
+env_file=".check.env"
 secret() { node -e 'console.log(require("crypto").randomBytes(+process.argv[1]).toString("base64url"))' "$1"; }
 port="${CHECK_PORT:-8088}"
 site="http://localhost:$port"
@@ -18,11 +20,14 @@ HTTP_PORT=$port
 HTTPS_PORT=${CHECK_TLS_PORT:-8443}
 POSTGRES_PASSWORD=$(secret 18)
 BETTER_AUTH_SECRET=$(secret 32)
+BACKUP_DIR=./.check-backups
 EOF
 compose() { docker compose -p "$project" --env-file "$env_file" "$@"; }
 cleanup() {
+  # Backup files belong to the container's user: remove them from inside.
+  compose run --rm --no-deps --entrypoint sh backup -c 'rm -rf /backups/*' >/dev/null 2>&1 || true
   compose down -v --remove-orphans >/dev/null 2>&1 || true
-  rm -f "$env_file"
+  rm -rf "$env_file" .check-backups
 }
 trap cleanup EXIT
 
@@ -65,4 +70,30 @@ expect "guest sign-up" "$(post /api/guest '{}')" '*"tutorial":{*'
 contract="$(curl -s -b "$jar" "$site/api/tutorial/board" | node -e 'let d="";process.stdin.on("data",c=>d+=c).on("end",()=>console.log(JSON.parse(d).contracts[0].id))')"
 expect "tutorial run dispatched" "$(post /api/dispatches "{\"kind\":\"tutorial\",\"contractId\":\"$contract\"}")" '*"dispatch":{*'
 
+# Step 13.3: back up, restore the latest dump into a fresh database, and find
+# that player's state exactly as it is in the live one.
+player="$(curl -s -b "$jar" "$site/api/state" | node -e 'let d="";process.stdin.on("data",c=>d+=c).on("end",()=>console.log(JSON.parse(d).playerId))')"
+expect "backup written" "$(compose exec -T backup /scripts/backup.sh)" 'backup: wrote /backups/deadhead-*.dump*'
+expect "latest dump restored" "$(compose exec -T backup /scripts/restore.sh)" 'restore: /backups/deadhead-*.dump -> deadhead_restore'
+player_state() {
+  local rows="" table column
+  for spec in players:id companies:player_id ships:player_id pilots:player_id bays:player_id \
+    discovered_locations:player_id contract_boards:player_id dispatches:player_id \
+    accepted_contracts:player_id; do
+    table="${spec%%:*}" column="${spec##*:}"
+    rows+="$table=$(compose exec -T backup psql -d "$1" -tAc \
+      "SELECT coalesce(json_agg(t ORDER BY t::text), '[]') FROM $table t WHERE $column = '$player'");"
+  done
+  echo "$rows"
+}
+live="$(player_state deadhead)"
+restored="$(player_state deadhead_restore)"
+expect "player found in the live database" "$live" '*"tutorial":true*'
+if [[ "$restored" == "$live" ]]; then
+  echo "ok   restored player state matches the live one"
+else
+  echo "FAIL restored player state differs"
+  fail=1
+fi
+
 exit "$fail"

deploy/compose.yml

@@ -1,5 +1,5 @@
-# Production stack for a home server (Phase 13): Postgres, the API server, and
+# Production stack for a home server (Phase 13): Postgres, the API server,
-# Caddy serving the web app with /api proxied to the server.
+# Caddy serving the web app with /api proxied to the server, and nightly backups.
 # From this folder:  docker compose --env-file .env up -d --build
 # Settings come from deploy/.env (copy .env.example).
 name: deadhead
@@ -55,6 +55,25 @@ services:
       - caddy-data:/data
       - caddy-config:/config
 
+  backup:
+    image: postgres:18
+    restart: unless-stopped
+    depends_on:
+      postgres:
+        condition: service_healthy
+    entrypoint: ["/bin/sh", "/scripts/schedule.sh"]
+    environment:
+      PGHOST: postgres
+      PGUSER: ${POSTGRES_USER:-deadhead}
+      PGPASSWORD: ${POSTGRES_PASSWORD}
+      PGDATABASE: ${POSTGRES_DB:-deadhead}
+      BACKUP_HOUR: ${BACKUP_HOUR:-3}
+      BACKUP_KEEP_DAYS: ${BACKUP_KEEP_DAYS:-14}
+      TZ: ${TZ:-UTC}
+    volumes:
+      - ./backup:/scripts:ro
+      - ${BACKUP_DIR:?set BACKUP_DIR in deploy/.env}:/backups
+
 volumes:
   postgres-data:
   caddy-data:

memory-bank/architecture.md

@@ -457,9 +457,10 @@ The app clock is `base clock + offset`; the offset is set only by the test-only
 | File | Role |
 |---|---|
 | `server.Dockerfile` | API image: production dependencies via `pnpm deploy`, TypeScript run with tsx, migrations then listen; non-root; health check on `/api/health` |
-| `compose.yml` | Production stack (project `deadhead`): Postgres 18 (named volume, not published), the server, and `web` (Caddy on `HTTP_PORT`/`HTTPS_PORT`, certificate volumes) |
+| `compose.yml` | Production stack (project `deadhead`): Postgres 18 (named volume, not published), the server, `web` (Caddy on `HTTP_PORT`/`HTTPS_PORT`, certificate volumes) and `backup` (`BACKUP_DIR` mounted at `/backups`) |
 | `web.Dockerfile`, `Caddyfile` | Caddy image with the built app: `/api/*` to `server:3000`, client routes fall back to `index.html`, hashed assets cached for a year, shell and service worker never; HTTPS for `SITE_ADDRESS` |
-| `README.md` | Home-server setup: DNS, port forwarding, `.env`, start/update, LAN-only mode |
+| `backup/` | `schedule.sh` (daily at `BACKUP_HOUR`), `backup.sh` (`pg_dump` to `BACKUP_DIR`, retention `BACKUP_KEEP_DAYS`), `restore.sh` (a dump into a database, default the newest into `deadhead_restore`); run by the `backup` service (Postgres image) |
+| `README.md` | Home-server setup: DNS, port forwarding, `.env`, start/update, LAN-only mode, backups and restores |
 | `.env.example` | `PUBLIC_URL` (also Better Auth's URL), `SITE_ADDRESS`, host ports, Postgres credentials, `BETTER_AUTH_SECRET`, log level; copy to `deploy/.env` |
 | `check.sh` | Builds and starts the stack under a throwaway project, checks it, removes it |
 

memory-bank/progress.md

@@ -323,4 +323,6 @@ Choices (asked at the start of the phase, decision 0.5.86): a home server runnin
 
 - [x] **Step 13.1 — Containerize the server.** `deploy/server.Dockerfile` (build from the repo root): installs the server's production dependencies with pnpm, makes a self-contained copy with `pnpm deploy`, and runs the TypeScript with tsx (now a server runtime dependency) as a non-root user; migrations run on every start, then the server listens; a health check calls `/api/health`. `deploy/compose.yml` (project `deadhead`): Postgres 18 with a named volume, not published, and the server, which waits for Postgres to be healthy. Settings come from `deploy/.env` (`deploy/.env.example`: `PUBLIC_URL`, Postgres user/db/password, `BETTER_AUTH_SECRET`, log level). `.dockerignore` keeps local state and the source art out of builds. Test: `deploy/check.sh` builds and starts the stack under a throwaway project with generated secrets, waits until it is healthy, checks `/api/health` and `/api/time` from inside the server container, and removes everything (passes, ~25 s with a warm cache).
 - [x] **Step 13.2 — Deploy the client.** `deploy/web.Dockerfile` builds the web app (`vite build` from the committed map art, `APP_BUILD` for the About screen's build) into a Caddy image. `deploy/Caddyfile`: one site at `SITE_ADDRESS` (the domain, so Let's Encrypt HTTPS; `:80` for plain HTTP), `/api/*` proxied to the server, everything else from the build with client routes falling back to `index.html`; Vite's hashed bundles and fonts cached for a year, the shell, service worker and manifest never. Compose gains the `web` service (ports from `HTTP_PORT`/`HTTPS_PORT`, certificate volumes). `deploy/README.md` covers DNS, port forwarding, settings, start/update and a LAN-only setup. Test: `deploy/check.sh` now also checks, through Caddy from the host: the app shell, a client route, `/api` through the proxy, the caching headers, and a guest sign-up plus a tutorial run dispatched (all pass). **Manual check for the user:** deploy on the home server and confirm the site loads on desktop and a phone, a guest can start, and a tutorial run can be dispatched.
-- [ ] Step 13.3 — Database backups
+- [x] **Step 13.3 — Database backups.** A `backup` service (the Postgres 18 image, scripts in `deploy/backup/`): `schedule.sh` runs `backup.sh` daily at `BACKUP_HOUR`:00 (`TZ`), which writes `deadhead-<UTC stamp>.dump` (`pg_dump` custom format, via a `.partial` name) into `BACKUP_DIR` (a host folder, e.g. a NAS mount) and deletes dumps older than `BACKUP_KEEP_DAYS` (default 14). `restore.sh [dump] [database]` loads a dump (default: the newest, into a new `deadhead_restore` database). README: backups, a test restore, and recovering the live database. Test: `deploy/check.sh` takes a backup of the running stack, restores the newest dump into a fresh database, and compares the test guest's rows (player, company, ships, pilots, bays, discoveries, boards, dispatches, accepted contracts) between the live and restored databases: identical (13/13 checks pass). 
+
+**Phase 13 complete — waiting for review.** Manual checks for the user: deploy on the home server (DNS, ports 80/443, `deploy/.env`), load the site on desktop and a phone, start a guest and dispatch a tutorial run; confirm a dump appears on the NAS after the first night.