From bd89c4cb58d91392428e5efa9b10d770b8baa381 Mon Sep 17 00:00:00 2001 From: fireshaper Date: Tue, 6 Oct 2026 17:53:54 -0400 Subject: [PATCH] Step 13.3: nightly database backups Co-Authored-By: Claude Opus 5.5 --- .dockerignore | 2 ++ .gitignore | 5 +++++ DECISIONS.md | 10 ++++++++++ deploy/.env.example | 7 +++++++ deploy/README.md | 37 ++++++++++++++++++++++++++++++++++--- deploy/backup/backup.sh | 13 +++++++++++++ deploy/backup/restore.sh | 15 +++++++++++++++ deploy/backup/schedule.sh | 13 +++++++++++++ deploy/check.sh | 35 +++++++++++++++++++++++++++++++++-- deploy/compose.yml | 23 +++++++++++++++++++++-- memory-bank/architecture.md | 5 +++-- memory-bank/progress.md | 4 +++- 12 files changed, 159 insertions(+), 10 deletions(-) create mode 100755 deploy/backup/backup.sh create mode 100755 deploy/backup/restore.sh create mode 100755 deploy/backup/schedule.sh diff --git a/.dockerignore b/.dockerignore index e703dc0..4fb266e 100644 --- a/.dockerignore +++ b/.dockerignore @@ -12,5 +12,7 @@ memory-bank docs deploy/backups +deploy/.check-backups +deploy/.check.env # The source map art is only an input to the asset script (decision 0.5.37). apps/web/public/assets/map diff --git a/.gitignore b/.gitignore index 91fd35b..bc26312 100644 --- a/.gitignore +++ b/.gitignore @@ -9,3 +9,8 @@ coverage/ # Playwright output apps/web/test-results/ apps/web/playwright-report/ + +# Local backup output (deploy/compose.yml) +deploy/backups/ +deploy/.check-backups/ +deploy/.check.env diff --git a/DECISIONS.md b/DECISIONS.md index 4c0901c..c8c9a3a 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -1315,3 +1315,13 @@ Implementation decisions not spelled out in `memory-bank/implementation-plan.md` - **`APP_BUILD` overrides the git commit in the build info**, because image builds have no `.git`; the README's update command passes the checkout's short hash. - **Caching:** only Vite's content-hashed bundles and fonts are marked immutable. The generated map art keeps its plain names, so it isn't; the service worker precache versions it anyway. The shell, service worker, its registration script and the manifest are `no-cache`, so a new deploy is picked up on the next visit (auto-update, Step 11.1). - **The 13.2 manual check is scripted locally** (guest sign-up and a tutorial run through Caddy, with the browser's Origin header); loading the real site on desktop and a phone is left to the user, since it needs the home server, DNS and port forwarding. + +## Step 13.3 + +- **Backups run in the stack** as a `backup` service on the same Postgres image (so `pg_dump` matches the server version), with a small shell scheduler rather than cron: one process, logs in `docker compose logs backup`, nothing to install on the host. +- **Off-server storage is a host folder** (`BACKUP_DIR`), as chosen: a NAS share mounted on the server. The README warns that an unmounted share would put dumps on the server's own disk. +- **Retention is age-based**: dumps older than `BACKUP_KEEP_DAYS` (default 14) are deleted after each successful backup, so a failing backup never deletes the last good ones by itself running. +- **Dumps use pg_dump's custom format** (compressed, restorable with `pg_restore --clean --if-exists --no-owner`), written under a `.partial` name and renamed when complete. +- **`restore.sh` defaults to a separate `deadhead_restore` database**, so the routine test restore never touches live data; recovering the live database is a documented, explicit command with the server stopped. +- **The step's test compares a player row by row**: after a guest's sign-up and tutorial run, the check backs up, restores into a fresh database and compares that player's rows across nine tables as JSON; they match exactly. +- **`check.sh` disables Git Bash's path conversion** (`MSYS_NO_PATHCONV`) so container paths reach Docker unchanged on Windows; its temporary settings file sits next to the compose file (`deploy/.check.env`, git- and docker-ignored) for the same reason. diff --git a/deploy/.env.example b/deploy/.env.example index c6ccdfe..11c0af0 100644 --- a/deploy/.env.example +++ b/deploy/.env.example @@ -19,3 +19,10 @@ POSTGRES_PASSWORD= BETTER_AUTH_SECRET= LOG_LEVEL=info + +# Nightly database backups: a folder on this machine, e.g. a NAS share mounted +# here. Dumps older than BACKUP_KEEP_DAYS are deleted. +BACKUP_DIR=/mnt/nas/deadhead-backups +BACKUP_HOUR=3 +BACKUP_KEEP_DAYS=14 +TZ=UTC diff --git a/deploy/README.md b/deploy/README.md index 267a18a..c5a6a32 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -41,6 +41,37 @@ deploy/check.sh ``` This builds and starts the whole stack under a separate project name with -throwaway secrets, on port 8088. It checks the API, the app, caching and a guest -sign-up with a tutorial run, then removes everything. It never touches the real -stack. +throwaway secrets, on port 8088. It checks the API, the app, caching, a guest +sign-up with a tutorial run, and a backup restored into a fresh database. Then +it removes everything. It never touches the real stack. + +## Backups + +The `backup` service writes a compressed dump of the database every night at +`BACKUP_HOUR`:00 (`TZ`, default UTC) to `BACKUP_DIR`, and deletes dumps older +than `BACKUP_KEEP_DAYS`. Point `BACKUP_DIR` at a folder that lives on another +disk, such as a NAS share mounted on the server (for example through +`/etc/fstab`). Make sure the share is mounted before Docker starts, or the +dumps land on the server's own disk. + +Take a backup now: + +```sh +docker compose exec backup /scripts/backup.sh +``` + +Test a restore without touching the live data. This loads the newest dump into +a separate `deadhead_restore` database: + +```sh +docker compose exec backup /scripts/restore.sh +``` + +Recover the live database from a dump (stop the server first so nothing writes +while it loads): + +```sh +docker compose stop server +docker compose exec backup /scripts/restore.sh /backups/deadhead-.dump deadhead +docker compose start server +``` diff --git a/deploy/backup/backup.sh b/deploy/backup/backup.sh new file mode 100755 index 0000000..ca25aa3 --- /dev/null +++ b/deploy/backup/backup.sh @@ -0,0 +1,13 @@ +#!/bin/sh +# One backup now (Step 13.3): a compressed pg_dump of the game database into +# /backups, then dumps older than BACKUP_KEEP_DAYS are deleted. Connection +# settings come from the PG* environment variables. +set -eu + +stamp="$(date -u +%Y%m%dT%H%M%SZ)" +file="/backups/deadhead-$stamp.dump" +# Written under a temporary name, so a half-written dump never looks complete. +pg_dump --format=custom --file="$file.partial" +mv "$file.partial" "$file" +find /backups -maxdepth 1 -name 'deadhead-*.dump' -mtime +"${BACKUP_KEEP_DAYS:-14}" -delete +echo "backup: wrote $file ($(du -h "$file" | cut -f1))" diff --git a/deploy/backup/restore.sh b/deploy/backup/restore.sh new file mode 100755 index 0000000..425a6eb --- /dev/null +++ b/deploy/backup/restore.sh @@ -0,0 +1,15 @@ +#!/bin/sh +# Restores a dump into a database (Step 13.3): +# restore.sh [dump] [database] +# Defaults: the newest dump in /backups, into a new database "deadhead_restore" +# (created if missing), so the live database is untouched. To replace the live +# database, stop the server first and pass its name. +set -eu + +dump="${1:-$(ls -1 /backups/deadhead-*.dump | sort | tail -n 1)}" +database="${2:-deadhead_restore}" +if ! psql -d postgres -tAc "SELECT 1 FROM pg_database WHERE datname = '$database'" | grep -q 1; then + createdb "$database" +fi +pg_restore --clean --if-exists --no-owner --dbname="$database" "$dump" +echo "restore: $dump -> $database" diff --git a/deploy/backup/schedule.sh b/deploy/backup/schedule.sh new file mode 100755 index 0000000..ac5f5d6 --- /dev/null +++ b/deploy/backup/schedule.sh @@ -0,0 +1,13 @@ +#!/bin/sh +# Runs backup.sh every day at BACKUP_HOUR:00 (in TZ, default UTC). +set -eu + +hour="${BACKUP_HOUR:-3}" +echo "backup: daily at ${hour}:00 ${TZ:-UTC}, keeping ${BACKUP_KEEP_DAYS:-14} days, into /backups" +while true; do + now="$(date +%s)" + next="$(date -d "today ${hour}:00" +%s)" + if [ "$next" -le "$now" ]; then next="$(date -d "tomorrow ${hour}:00" +%s)"; fi + sleep "$((next - now))" + /scripts/backup.sh || echo "backup: FAILED" >&2 +done diff --git a/deploy/check.sh b/deploy/check.sh index 1bd7042..ab37c97 100755 --- a/deploy/check.sh +++ b/deploy/check.sh @@ -4,10 +4,12 @@ # deploy/check.sh from the repository root or deploy/ # Exits non-zero if any check fails. Needs Docker with Compose v2. set -euo pipefail +# Git Bash on Windows would rewrite container paths like /scripts/… otherwise. +export MSYS_NO_PATHCONV=1 cd "$(dirname "$0")" project="deadhead-check" -env_file="$(mktemp)" +env_file=".check.env" secret() { node -e 'console.log(require("crypto").randomBytes(+process.argv[1]).toString("base64url"))' "$1"; } port="${CHECK_PORT:-8088}" site="http://localhost:$port" @@ -18,11 +20,14 @@ HTTP_PORT=$port HTTPS_PORT=${CHECK_TLS_PORT:-8443} POSTGRES_PASSWORD=$(secret 18) BETTER_AUTH_SECRET=$(secret 32) +BACKUP_DIR=./.check-backups EOF compose() { docker compose -p "$project" --env-file "$env_file" "$@"; } cleanup() { + # Backup files belong to the container's user: remove them from inside. + compose run --rm --no-deps --entrypoint sh backup -c 'rm -rf /backups/*' >/dev/null 2>&1 || true compose down -v --remove-orphans >/dev/null 2>&1 || true - rm -f "$env_file" + rm -rf "$env_file" .check-backups } trap cleanup EXIT @@ -65,4 +70,30 @@ expect "guest sign-up" "$(post /api/guest '{}')" '*"tutorial":{*' contract="$(curl -s -b "$jar" "$site/api/tutorial/board" | node -e 'let d="";process.stdin.on("data",c=>d+=c).on("end",()=>console.log(JSON.parse(d).contracts[0].id))')" expect "tutorial run dispatched" "$(post /api/dispatches "{\"kind\":\"tutorial\",\"contractId\":\"$contract\"}")" '*"dispatch":{*' +# Step 13.3: back up, restore the latest dump into a fresh database, and find +# that player's state exactly as it is in the live one. +player="$(curl -s -b "$jar" "$site/api/state" | node -e 'let d="";process.stdin.on("data",c=>d+=c).on("end",()=>console.log(JSON.parse(d).playerId))')" +expect "backup written" "$(compose exec -T backup /scripts/backup.sh)" 'backup: wrote /backups/deadhead-*.dump*' +expect "latest dump restored" "$(compose exec -T backup /scripts/restore.sh)" 'restore: /backups/deadhead-*.dump -> deadhead_restore' +player_state() { + local rows="" table column + for spec in players:id companies:player_id ships:player_id pilots:player_id bays:player_id \ + discovered_locations:player_id contract_boards:player_id dispatches:player_id \ + accepted_contracts:player_id; do + table="${spec%%:*}" column="${spec##*:}" + rows+="$table=$(compose exec -T backup psql -d "$1" -tAc \ + "SELECT coalesce(json_agg(t ORDER BY t::text), '[]') FROM $table t WHERE $column = '$player'");" + done + echo "$rows" +} +live="$(player_state deadhead)" +restored="$(player_state deadhead_restore)" +expect "player found in the live database" "$live" '*"tutorial":true*' +if [[ "$restored" == "$live" ]]; then + echo "ok restored player state matches the live one" +else + echo "FAIL restored player state differs" + fail=1 +fi + exit "$fail" diff --git a/deploy/compose.yml b/deploy/compose.yml index 2e1f40b..2e4a5d2 100644 --- a/deploy/compose.yml +++ b/deploy/compose.yml @@ -1,5 +1,5 @@ -# Production stack for a home server (Phase 13): Postgres, the API server, and -# Caddy serving the web app with /api proxied to the server. +# Production stack for a home server (Phase 13): Postgres, the API server, +# Caddy serving the web app with /api proxied to the server, and nightly backups. # From this folder: docker compose --env-file .env up -d --build # Settings come from deploy/.env (copy .env.example). name: deadhead @@ -55,6 +55,25 @@ services: - caddy-data:/data - caddy-config:/config + backup: + image: postgres:18 + restart: unless-stopped + depends_on: + postgres: + condition: service_healthy + entrypoint: ["/bin/sh", "/scripts/schedule.sh"] + environment: + PGHOST: postgres + PGUSER: ${POSTGRES_USER:-deadhead} + PGPASSWORD: ${POSTGRES_PASSWORD} + PGDATABASE: ${POSTGRES_DB:-deadhead} + BACKUP_HOUR: ${BACKUP_HOUR:-3} + BACKUP_KEEP_DAYS: ${BACKUP_KEEP_DAYS:-14} + TZ: ${TZ:-UTC} + volumes: + - ./backup:/scripts:ro + - ${BACKUP_DIR:?set BACKUP_DIR in deploy/.env}:/backups + volumes: postgres-data: caddy-data: diff --git a/memory-bank/architecture.md b/memory-bank/architecture.md index c2cfd0b..d42562b 100644 --- a/memory-bank/architecture.md +++ b/memory-bank/architecture.md @@ -457,9 +457,10 @@ The app clock is `base clock + offset`; the offset is set only by the test-only | File | Role | |---|---| | `server.Dockerfile` | API image: production dependencies via `pnpm deploy`, TypeScript run with tsx, migrations then listen; non-root; health check on `/api/health` | -| `compose.yml` | Production stack (project `deadhead`): Postgres 18 (named volume, not published), the server, and `web` (Caddy on `HTTP_PORT`/`HTTPS_PORT`, certificate volumes) | +| `compose.yml` | Production stack (project `deadhead`): Postgres 18 (named volume, not published), the server, `web` (Caddy on `HTTP_PORT`/`HTTPS_PORT`, certificate volumes) and `backup` (`BACKUP_DIR` mounted at `/backups`) | | `web.Dockerfile`, `Caddyfile` | Caddy image with the built app: `/api/*` to `server:3000`, client routes fall back to `index.html`, hashed assets cached for a year, shell and service worker never; HTTPS for `SITE_ADDRESS` | -| `README.md` | Home-server setup: DNS, port forwarding, `.env`, start/update, LAN-only mode | +| `backup/` | `schedule.sh` (daily at `BACKUP_HOUR`), `backup.sh` (`pg_dump` to `BACKUP_DIR`, retention `BACKUP_KEEP_DAYS`), `restore.sh` (a dump into a database, default the newest into `deadhead_restore`); run by the `backup` service (Postgres image) | +| `README.md` | Home-server setup: DNS, port forwarding, `.env`, start/update, LAN-only mode, backups and restores | | `.env.example` | `PUBLIC_URL` (also Better Auth's URL), `SITE_ADDRESS`, host ports, Postgres credentials, `BETTER_AUTH_SECRET`, log level; copy to `deploy/.env` | | `check.sh` | Builds and starts the stack under a throwaway project, checks it, removes it | diff --git a/memory-bank/progress.md b/memory-bank/progress.md index 7d514d6..4019dc9 100644 --- a/memory-bank/progress.md +++ b/memory-bank/progress.md @@ -323,4 +323,6 @@ Choices (asked at the start of the phase, decision 0.5.86): a home server runnin - [x] **Step 13.1 — Containerize the server.** `deploy/server.Dockerfile` (build from the repo root): installs the server's production dependencies with pnpm, makes a self-contained copy with `pnpm deploy`, and runs the TypeScript with tsx (now a server runtime dependency) as a non-root user; migrations run on every start, then the server listens; a health check calls `/api/health`. `deploy/compose.yml` (project `deadhead`): Postgres 18 with a named volume, not published, and the server, which waits for Postgres to be healthy. Settings come from `deploy/.env` (`deploy/.env.example`: `PUBLIC_URL`, Postgres user/db/password, `BETTER_AUTH_SECRET`, log level). `.dockerignore` keeps local state and the source art out of builds. Test: `deploy/check.sh` builds and starts the stack under a throwaway project with generated secrets, waits until it is healthy, checks `/api/health` and `/api/time` from inside the server container, and removes everything (passes, ~25 s with a warm cache). - [x] **Step 13.2 — Deploy the client.** `deploy/web.Dockerfile` builds the web app (`vite build` from the committed map art, `APP_BUILD` for the About screen's build) into a Caddy image. `deploy/Caddyfile`: one site at `SITE_ADDRESS` (the domain, so Let's Encrypt HTTPS; `:80` for plain HTTP), `/api/*` proxied to the server, everything else from the build with client routes falling back to `index.html`; Vite's hashed bundles and fonts cached for a year, the shell, service worker and manifest never. Compose gains the `web` service (ports from `HTTP_PORT`/`HTTPS_PORT`, certificate volumes). `deploy/README.md` covers DNS, port forwarding, settings, start/update and a LAN-only setup. Test: `deploy/check.sh` now also checks, through Caddy from the host: the app shell, a client route, `/api` through the proxy, the caching headers, and a guest sign-up plus a tutorial run dispatched (all pass). **Manual check for the user:** deploy on the home server and confirm the site loads on desktop and a phone, a guest can start, and a tutorial run can be dispatched. -- [ ] Step 13.3 — Database backups +- [x] **Step 13.3 — Database backups.** A `backup` service (the Postgres 18 image, scripts in `deploy/backup/`): `schedule.sh` runs `backup.sh` daily at `BACKUP_HOUR`:00 (`TZ`), which writes `deadhead-.dump` (`pg_dump` custom format, via a `.partial` name) into `BACKUP_DIR` (a host folder, e.g. a NAS mount) and deletes dumps older than `BACKUP_KEEP_DAYS` (default 14). `restore.sh [dump] [database]` loads a dump (default: the newest, into a new `deadhead_restore` database). README: backups, a test restore, and recovering the live database. Test: `deploy/check.sh` takes a backup of the running stack, restores the newest dump into a fresh database, and compares the test guest's rows (player, company, ships, pilots, bays, discoveries, boards, dispatches, accepted contracts) between the live and restored databases: identical (13/13 checks pass). + +**Phase 13 complete — waiting for review.** Manual checks for the user: deploy on the home server (DNS, ports 80/443, `deploy/.env`), load the site on desktop and a phone, start a guest and dispatch a tutorial run; confirm a dump appears on the NAS after the first night.