Add a health check and the deployment runbook
/healthz checks the database and does a cache ROUND TRIP, not a ping. Both matter: a node that cannot reach Postgres serves nothing, and a cache that accepts writes and returns nothing would have waffle read every feature flag as unset -- so "healthy" has to mean more than "the process is listening", or the load balancer will keep feeding traffic to a node that only looks alive. No auth and no tenant on it: the proxy, and later a load balancer, must reach it on any host. DEPLOYMENT.md is the runbook, and leads with the five things that make this app not a generic Django deploy: the wildcard cert forces DNS-01 (Let's Encrypt will not issue a wildcard over HTTP-01); Redis is required on one server, not two, because of the per-process flag cache; SECURE_PROXY_SSL_HEADER plus Caddy's X-Forwarded-Proto or WebAuthn and the SSL redirect both break; uploads must reach object storage BEFORE the second app server, not during; and invoices need native pango. Also documents why the archive job ships with --commit off, why migrations are run explicitly rather than from the entrypoint, and how to test the restore before the day you need it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
148
DEPLOYMENT.md
Normal file
148
DEPLOYMENT.md
Normal file
@@ -0,0 +1,148 @@
|
||||
# Deploying RosterChief
|
||||
|
||||
One server today, several later, with no code changes in between — only environment
|
||||
variables. This document is the runbook and, more usefully, the list of things that are
|
||||
specific to *this* app and will bite you if you treat it as a generic Django deploy.
|
||||
|
||||
## The five things that make this deployment unusual
|
||||
|
||||
**1. You need a wildcard TLS certificate, and that forces DNS-01.**
|
||||
Tenancy is subdomain-based (`ajax.rosterchief.app`), so the certificate must cover
|
||||
`*.rosterchief.app`. Let's Encrypt **will not issue a wildcard over HTTP-01** — only over
|
||||
DNS-01, which means the TLS terminator needs API access to your DNS zone. That is why
|
||||
`deploy/caddy/Dockerfile` builds Caddy *with* a DNS provider plugin, and why
|
||||
`CLOUDFLARE_API_TOKEN` is a required variable rather than a nicety. Swap the plugin
|
||||
(`caddy-dns/route53`, `caddy-dns/digitalocean`, …) if your DNS lives elsewhere.
|
||||
|
||||
DNS needs two records, both pointing at the server:
|
||||
|
||||
```
|
||||
A rosterchief.app -> <server ip>
|
||||
A *.rosterchief.app -> <server ip>
|
||||
```
|
||||
|
||||
**2. Redis is not optional, even on one server.**
|
||||
`waffle` caches each feature flag's targeting in the Django cache, and `LocMemCache` is
|
||||
private to a single process. Under several gunicorn workers, toggling a feature in the
|
||||
control panel flushes **one** worker's cache while the others keep serving the stale flag —
|
||||
a feature that "sometimes doesn't turn on". A shared cache is the fix.
|
||||
|
||||
**3. `SECURE_PROXY_SSL_HEADER` must be set, and Caddy must send the header.**
|
||||
Caddy terminates TLS, so without it Django believes every request is plain HTTP:
|
||||
`request.is_secure()` goes false, WebAuthn disagrees with the browser about the origin, and
|
||||
`SECURE_SSL_REDIRECT` becomes a redirect loop. Both halves are already wired (settings +
|
||||
`header_up X-Forwarded-Proto`); don't remove either.
|
||||
|
||||
**4. Uploads must move to object storage before the second app server.**
|
||||
Club logos go to `MEDIA_ROOT` on local disk. On one box that is fine. On two, a logo
|
||||
uploaded to node A is a 404 on node B. Setting `AWS_STORAGE_BUCKET_NAME` switches the
|
||||
default storage to S3 — do it *before* you scale, not during.
|
||||
|
||||
**5. PDF invoices need native libraries.**
|
||||
WeasyPrint binds to pango/cairo. The image installs them; a bare-metal deploy would need
|
||||
them too, and a Mac needs Homebrew. This is the main reason to run the container even in
|
||||
development if you touch invoicing.
|
||||
|
||||
## First deploy
|
||||
|
||||
```bash
|
||||
# 1. Configure
|
||||
cp .env.compose.example .env # read by docker compose
|
||||
cp .env.production.example .env.production # read by Django
|
||||
python -c "import secrets; print(secrets.token_urlsafe(64))" # -> DJANGO_SECRET_KEY
|
||||
|
||||
# 2. Build and start
|
||||
docker compose build
|
||||
docker compose up -d db redis
|
||||
docker compose run --rm web python manage.py migrate
|
||||
docker compose run --rm web python manage.py createsuperuser
|
||||
docker compose up -d
|
||||
|
||||
# 3. Verify
|
||||
curl -fsS https://rosterchief.app/healthz # {"status": "ok", ...}
|
||||
docker compose run --rm web python manage.py check --deploy
|
||||
```
|
||||
|
||||
`check --deploy` is what catches an env file that forgot the HTTPS flags: they default to
|
||||
**off** in code, because defaulting them to `not DEBUG` would redirect every test request to
|
||||
https and break the suite anywhere `DEBUG` is unset.
|
||||
|
||||
The first `docker compose up` will take a minute or two: Caddy is provisioning the wildcard
|
||||
certificate over DNS-01, and DNS propagation is not instant. Watch it with
|
||||
`docker compose logs -f caddy`.
|
||||
|
||||
## Migrations
|
||||
|
||||
Deliberately **not** run by the container's entrypoint. With more than one web container they
|
||||
would race, and a starting gunicorn worker is a bad place to discover a failed migration.
|
||||
Run them once, explicitly, as part of the deploy:
|
||||
|
||||
```bash
|
||||
docker compose build
|
||||
docker compose run --rm web python manage.py migrate
|
||||
docker compose up -d --no-deps web
|
||||
```
|
||||
|
||||
## Scheduled jobs
|
||||
|
||||
Two commands need to run on a schedule. Put them on the **host**, not in a container, and on
|
||||
**exactly one node** when you have several — three nodes archiving the same club is three
|
||||
emails to the same club.
|
||||
|
||||
```cron
|
||||
# Bill: archive clubs unpaid past their grace period.
|
||||
# Run it WITHOUT --commit for the first week and read the output. The flag exists because
|
||||
# this switches off paying customers: a bad clock or a bad cron should cost you an email,
|
||||
# not a morning of angry clubs.
|
||||
0 6 * * * cd /srv/rosterchief && docker compose run --rm web python manage.py archive_overdue_clubs --commit
|
||||
|
||||
# Events: extend recurring series so the calendar never runs dry.
|
||||
0 3 * * * cd /srv/rosterchief && docker compose run --rm web python manage.py extend_event_series
|
||||
```
|
||||
|
||||
## Backups
|
||||
|
||||
Two things carry state: Postgres and the uploads.
|
||||
|
||||
```bash
|
||||
# Database
|
||||
docker compose exec -T db pg_dump -U rosterchief rosterchief | gzip > rosterchief-$(date +%F).sql.gz
|
||||
|
||||
# Uploads — until they are on S3, in which case the bucket's own versioning is the backup.
|
||||
docker compose cp web:/app/media ./media-backup
|
||||
```
|
||||
|
||||
Restore is `gunzip -c dump.sql.gz | docker compose exec -T db psql -U rosterchief rosterchief`.
|
||||
Test it once, now, rather than the first time you need it.
|
||||
|
||||
## Going multi-server
|
||||
|
||||
Nothing in the code changes. What changes is where the services live:
|
||||
|
||||
| | one server | several |
|
||||
|---|---|---|
|
||||
| Postgres | `db` container | `DJANGO_DATABASE_URL` → your central Postgres |
|
||||
| Cache / flags | `redis` container | managed Redis (or your existing one) |
|
||||
| Uploads | local disk | **S3 bucket** (`AWS_STORAGE_BUCKET_NAME`) |
|
||||
| Static files | WhiteNoise, in the image | unchanged — that is why WhiteNoise is there |
|
||||
| Cron | host crontab | one node only |
|
||||
| TLS | Caddy on the box | load balancer, or Caddy on each node |
|
||||
|
||||
Drop `db` and `redis` from `compose.yaml`, point the URLs at the central services, and run
|
||||
`web` on as many nodes as you like behind a load balancer pointed at `/healthz`.
|
||||
|
||||
The health check tests the database *and* a cache round trip, not just that the process is
|
||||
listening — a node that cannot reach Postgres, or whose cache silently swallows writes, is
|
||||
not healthy, and a load balancer must not keep feeding it traffic.
|
||||
|
||||
## Rollback
|
||||
|
||||
Images are the unit of rollback. Tag on build, keep the last few, and:
|
||||
|
||||
```bash
|
||||
docker compose up -d --no-deps web # with the previous image tag
|
||||
```
|
||||
|
||||
Migrations are the exception: they don't roll back with the image. Prefer additive migrations
|
||||
(add a column, deploy, backfill, then stop writing the old one) so that yesterday's image
|
||||
still runs against today's schema.
|
||||
44
rosterchief/health.py
Normal file
44
rosterchief/health.py
Normal file
@@ -0,0 +1,44 @@
|
||||
"""Liveness for the proxy today, for a load balancer later.
|
||||
|
||||
Checks the two dependencies whose absence makes the app lie rather than fail: without the
|
||||
database it cannot serve anything, and without a shared cache the feature flags drift apart
|
||||
between workers. A health check that only proves the process is listening would call that
|
||||
healthy.
|
||||
"""
|
||||
|
||||
from django.core.cache import cache
|
||||
from django.db import connection
|
||||
from django.http import JsonResponse
|
||||
from django.views.decorators.cache import never_cache
|
||||
|
||||
PROBE_KEY = "healthz"
|
||||
|
||||
|
||||
@never_cache
|
||||
def healthz(request):
|
||||
checks = {"database": _database(), "cache": _cache()}
|
||||
healthy = all(checks.values())
|
||||
|
||||
return JsonResponse({"status": "ok" if healthy else "degraded", "checks": checks}, status=200 if healthy else 503)
|
||||
|
||||
|
||||
def _database() -> bool:
|
||||
try:
|
||||
with connection.cursor() as cursor:
|
||||
cursor.execute("SELECT 1")
|
||||
cursor.fetchone()
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
|
||||
def _cache() -> bool:
|
||||
"""A round trip, not a ping: a cache that accepts writes and returns nothing is worse
|
||||
than one that is plainly down, because waffle would read every flag as unset."""
|
||||
try:
|
||||
cache.set(PROBE_KEY, "ok", 10)
|
||||
|
||||
return cache.get(PROBE_KEY) == "ok"
|
||||
except Exception:
|
||||
return False
|
||||
@@ -1,7 +1,9 @@
|
||||
import importlib
|
||||
from unittest import mock
|
||||
|
||||
from django.db.utils import OperationalError
|
||||
from django.test import SimpleTestCase, override_settings
|
||||
from django.urls import Resolver404, clear_url_caches, resolve
|
||||
from django.urls import Resolver404, clear_url_caches, resolve, reverse
|
||||
|
||||
from . import urls
|
||||
|
||||
@@ -28,3 +30,51 @@ class BrowserReloadUrlTests(SimpleTestCase):
|
||||
|
||||
with self.assertRaises(Resolver404):
|
||||
resolve("/__reload__/events/")
|
||||
|
||||
|
||||
class HealthCheckTests(SimpleTestCase):
|
||||
databases = {"default"}
|
||||
|
||||
def test_it_reports_ok_when_the_database_and_cache_answer(self):
|
||||
response = self.client.get(reverse("healthz"))
|
||||
|
||||
self.assertEqual(response.status_code, 200)
|
||||
self.assertEqual(response.json(), {"status": "ok", "checks": {"database": True, "cache": True}})
|
||||
|
||||
def test_a_dead_database_is_a_503(self):
|
||||
# 200 while the database is unreachable is worse than no health check at all: the load
|
||||
# balancer would keep sending traffic to a node that cannot serve a single page.
|
||||
with mock.patch("rosterchief.health.connection") as db:
|
||||
db.cursor.side_effect = OperationalError("connection refused")
|
||||
|
||||
response = self.client.get(reverse("healthz"))
|
||||
|
||||
self.assertEqual(response.status_code, 503)
|
||||
self.assertFalse(response.json()["checks"]["database"])
|
||||
|
||||
def test_a_cache_that_swallows_writes_is_unhealthy(self):
|
||||
# Not a ping: a cache that accepts writes and returns nothing would have waffle read
|
||||
# every feature flag as unset.
|
||||
with mock.patch("rosterchief.health.cache") as broken:
|
||||
broken.get.return_value = None
|
||||
|
||||
response = self.client.get(reverse("healthz"))
|
||||
|
||||
self.assertEqual(response.status_code, 503)
|
||||
self.assertFalse(response.json()["checks"]["cache"])
|
||||
|
||||
def test_an_unreachable_cache_is_a_503(self):
|
||||
# Distinct from the cache that answers wrongly above: here Redis is simply down, and
|
||||
# the check must fail rather than raise its way to a 500.
|
||||
with mock.patch("rosterchief.health.cache") as down:
|
||||
down.set.side_effect = ConnectionError("redis is down")
|
||||
|
||||
response = self.client.get(reverse("healthz"))
|
||||
|
||||
self.assertEqual(response.status_code, 503)
|
||||
self.assertFalse(response.json()["checks"]["cache"])
|
||||
|
||||
def test_it_is_never_cached(self):
|
||||
response = self.client.get(reverse("healthz"))
|
||||
|
||||
self.assertIn("no-cache", response["Cache-Control"])
|
||||
|
||||
@@ -15,7 +15,11 @@ from django.views.generic import RedirectView
|
||||
|
||||
from club.views import root
|
||||
|
||||
from .health import healthz
|
||||
|
||||
urlpatterns = [
|
||||
# No auth and no tenant: the proxy and the load balancer must reach it on any host.
|
||||
path("healthz", healthz, name="healthz"),
|
||||
path("admin/login/", RedirectView.as_view(pattern_name="account_login", query_string=True), name="admin_login_redirect"),
|
||||
path("admin/", admin.site.urls),
|
||||
path("accounts/", include("allauth.urls")),
|
||||
|
||||
Reference in New Issue
Block a user