Add a health check and the deployment runbook

/healthz checks the database and does a cache ROUND TRIP, not a ping. Both matter:
a node that cannot reach Postgres serves nothing, and a cache that accepts writes
and returns nothing would have waffle read every feature flag as unset -- so
"healthy" has to mean more than "the process is listening", or the load balancer
will keep feeding traffic to a node that only looks alive.

No auth and no tenant on it: the proxy, and later a load balancer, must reach it on
any host.

DEPLOYMENT.md is the runbook, and leads with the five things that make this app not
a generic Django deploy: the wildcard cert forces DNS-01 (Let's Encrypt will not
issue a wildcard over HTTP-01); Redis is required on one server, not two, because
of the per-process flag cache; SECURE_PROXY_SSL_HEADER plus Caddy's
X-Forwarded-Proto or WebAuthn and the SSL redirect both break; uploads must reach
object storage BEFORE the second app server, not during; and invoices need native
pango.

Also documents why the archive job ships with --commit off, why migrations are run
explicitly rather than from the entrypoint, and how to test the restore before the
day you need it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

rosterchief/health.py

@@ -0,0 +1,44 @@
+"""Liveness for the proxy today, for a load balancer later.
+
+Checks the two dependencies whose absence makes the app lie rather than fail: without the
+database it cannot serve anything, and without a shared cache the feature flags drift apart
+between workers. A health check that only proves the process is listening would call that
+healthy.
+"""
+
+from django.core.cache import cache
+from django.db import connection
+from django.http import JsonResponse
+from django.views.decorators.cache import never_cache
+
+PROBE_KEY = "healthz"
+
+
+@never_cache
+def healthz(request):
+    checks = {"database": _database(), "cache": _cache()}
+    healthy = all(checks.values())
+
+    return JsonResponse({"status": "ok" if healthy else "degraded", "checks": checks}, status=200 if healthy else 503)
+
+
+def _database() -> bool:
+    try:
+        with connection.cursor() as cursor:
+            cursor.execute("SELECT 1")
+            cursor.fetchone()
+    except Exception:
+        return False
+
+    return True
+
+
+def _cache() -> bool:
+    """A round trip, not a ping: a cache that accepts writes and returns nothing is worse
+    than one that is plainly down, because waffle would read every flag as unset."""
+    try:
+        cache.set(PROBE_KEY, "ok", 10)
+
+        return cache.get(PROBE_KEY) == "ok"
+    except Exception:
+        return False