Add maintenance mode: lock the platform down from the control panel

Closes every club subdomain with a 503 in that club's own colours, stands the
scheduled jobs down, and keeps open exactly what is needed to end it again.

The exemptions ARE the feature:

- /accounts/ stays open on the base domain. Close it too and you cannot sign in to
  turn maintenance off -- a lock-down with no key, fixable only from a shell.
- /healthz answers on every host. Close it and the load balancer decides the node
  is dead, stops routing to it, and takes the control panel down with everything
  else.
- migrate and collectstatic are NOT blocked. Maintenance is usually declared in
  order to run them; a blanket guard on BaseCommand would mean turning the mode off
  to do the work you turned it on for. Only the domain jobs (archive_overdue_clubs,
  extend_event_series, import_members_csv) refuse, and they exit non-zero so cron
  mails you -- a scheduled job that silently skips itself is how a month of billing
  goes missing.

The state is cached with a 10-second TTL, not for ever. Write-through makes the
flip instant for the shared Redis of a real deployment, and the TTL is the belt to
that braces: on a per-process cache -- a dev box with no Redis, or a misconfigured
deploy -- a lock-down that reached only one gunicorn worker would be worse than
useless. Live-verified: a club subdomain, its login page and the base domain all
503 while the control panel and the sign-in screens stay up.

Also adds the two deployment pieces asked for: compose.behind-proxy.yaml for a
dev/test box that already runs Caddy on :80 (app on the loopback, host Caddy proxies
to it -- and the host's Caddy still needs the DNS plugin, because the wildcard is
still a wildcard), and deploy/backup.sh + restore-check.sh with a cron schedule. The
backup writes to a .part file and only lands it once gzip -t says it is readable: a
truncated dump that looks like a backup is the failure you find on the day you need
it. The weekly restore rehearsal is the only line in that cron that proves the rest
work.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

deploy/backup.sh

@@ -0,0 +1,51 @@
+#!/usr/bin/env bash
+# Back up what carries state: the database, and the uploads if they are still on local disk.
+#
+#   deploy/backup.sh /var/backups/rosterchief
+#
+# Runs from cron (see DEPLOYMENT.md). Exits non-zero on any failure, so cron mails you —
+# a backup script that fails quietly is worse than no backup script, because you will
+# believe you have backups.
+set -Eeuo pipefail
+
+DEST="${1:-/var/backups/rosterchief}"
+COMPOSE="${COMPOSE:-docker compose}"
+KEEP_DAYS="${KEEP_DAYS:-14}"
+STAMP="$(date +%F-%H%M)"
+
+mkdir -p "$DEST"
+
+# --- database ---------------------------------------------------------------
+# Written to a temporary name and moved into place only on success: a truncated dump that
+# looks like a backup is the trap this avoids.
+DB_TMP="$DEST/.db-$STAMP.sql.gz.part"
+DB_OUT="$DEST/db-$STAMP.sql.gz"
+
+$COMPOSE exec -T db pg_dump --clean --if-exists -U "${POSTGRES_USER:-rosterchief}" "${POSTGRES_DB:-rosterchief}" | gzip > "$DB_TMP"
+gzip -t "$DB_TMP"                     # the archive is readable
+[ -s "$DB_TMP" ]                      # ...and not empty
+mv "$DB_TMP" "$DB_OUT"
+
+# --- uploads ----------------------------------------------------------------
+# Only while media is local. Once AWS_STORAGE_BUCKET_NAME is set the bucket's own versioning
+# is the backup, and this step is skipped.
+if [ -z "${AWS_STORAGE_BUCKET_NAME:-}" ]; then
+    MEDIA_OUT="$DEST/media-$STAMP.tar.gz"
+    $COMPOSE exec -T web tar -cz -C /app media | cat > "$MEDIA_OUT.part"
+    mv "$MEDIA_OUT.part" "$MEDIA_OUT"
+fi
+
+# --- retention --------------------------------------------------------------
+find "$DEST" -name 'db-*.sql.gz' -mtime "+$KEEP_DAYS" -delete
+find "$DEST" -name 'media-*.tar.gz' -mtime "+$KEEP_DAYS" -delete
+find "$DEST" -name '*.part' -mtime +1 -delete
+
+echo "$(date -Iseconds) backup ok: $(basename "$DB_OUT") ($(du -h "$DB_OUT" | cut -f1))"
+
+# --- offsite ----------------------------------------------------------------
+# A backup on the same disk as the database is not a backup: it survives a bad migration, but
+# not the server. Set BACKUP_REMOTE to an rclone remote to copy it off the box.
+if [ -n "${BACKUP_REMOTE:-}" ]; then
+    rclone copy "$DEST" "$BACKUP_REMOTE" --max-age "${KEEP_DAYS}d"
+    echo "$(date -Iseconds) copied to $BACKUP_REMOTE"
+fi

deploy/restore-check.sh

@@ -0,0 +1,30 @@
+#!/usr/bin/env bash
+# Restore the latest dump into a throwaway database and count the rows.
+#
+#   deploy/restore-check.sh [/var/backups/rosterchief]
+#
+# The only line in the backup cron that proves the others work. A dump you have never
+# restored is a hypothesis, not a backup.
+set -Eeuo pipefail
+
+DEST="${1:-/var/backups/rosterchief}"
+COMPOSE="${COMPOSE:-docker compose}"
+USER_NAME="${POSTGRES_USER:-rosterchief}"
+SCRATCH="restore_check_$(date +%s)"
+
+LATEST="$(ls -1t "$DEST"/db-*.sql.gz 2>/dev/null | head -1)"
+[ -n "$LATEST" ] || { echo "no dump found in $DEST"; exit 1; }
+
+cleanup() { $COMPOSE exec -T db dropdb -U "$USER_NAME" --if-exists "$SCRATCH" >/dev/null 2>&1 || true; }
+trap cleanup EXIT
+
+$COMPOSE exec -T db createdb -U "$USER_NAME" "$SCRATCH"
+gunzip -c "$LATEST" | $COMPOSE exec -T db psql -q -U "$USER_NAME" "$SCRATCH" >/dev/null
+
+# A restore that produces an empty schema exits 0 and tells you nothing. Ask it something.
+CLUBS="$($COMPOSE exec -T db psql -tAq -U "$USER_NAME" "$SCRATCH" -c 'SELECT count(*) FROM club_club')"
+USERS="$($COMPOSE exec -T db psql -tAq -U "$USER_NAME" "$SCRATCH" -c 'SELECT count(*) FROM authentication_user')"
+
+[ "$USERS" -gt 0 ] || { echo "restore check FAILED: $(basename "$LATEST") restored no users"; exit 1; }
+
+echo "$(date -Iseconds) restore ok: $(basename "$LATEST") -> $CLUBS clubs, $USERS users"