Add maintenance mode: lock the platform down from the control panel

Closes every club subdomain with a 503 in that club's own colours, stands the
scheduled jobs down, and keeps open exactly what is needed to end it again.

The exemptions ARE the feature:

- /accounts/ stays open on the base domain. Close it too and you cannot sign in to
  turn maintenance off -- a lock-down with no key, fixable only from a shell.
- /healthz answers on every host. Close it and the load balancer decides the node
  is dead, stops routing to it, and takes the control panel down with everything
  else.
- migrate and collectstatic are NOT blocked. Maintenance is usually declared in
  order to run them; a blanket guard on BaseCommand would mean turning the mode off
  to do the work you turned it on for. Only the domain jobs (archive_overdue_clubs,
  extend_event_series, import_members_csv) refuse, and they exit non-zero so cron
  mails you -- a scheduled job that silently skips itself is how a month of billing
  goes missing.

The state is cached with a 10-second TTL, not for ever. Write-through makes the
flip instant for the shared Redis of a real deployment, and the TTL is the belt to
that braces: on a per-process cache -- a dev box with no Redis, or a misconfigured
deploy -- a lock-down that reached only one gunicorn worker would be worse than
useless. Live-verified: a club subdomain, its login page and the base domain all
503 while the control panel and the sign-in screens stay up.

Also adds the two deployment pieces asked for: compose.behind-proxy.yaml for a
dev/test box that already runs Caddy on :80 (app on the loopback, host Caddy proxies
to it -- and the host's Caddy still needs the DNS plugin, because the wildcard is
still a wildcard), and deploy/backup.sh + restore-check.sh with a cron schedule. The
backup writes to a .part file and only lands it once gzip -t says it is readable: a
truncated dump that looks like a backup is the failure you find on the day you need
it. The weekly restore rehearsal is the only line in that cron that proves the rest
work.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

features/commands.py

@@ -0,0 +1,31 @@
+"""Scheduled work refuses to run while the platform is locked down.
+
+Deliberately opt-in, per command, rather than a blanket guard on BaseCommand: maintenance is
+usually declared IN ORDER to run `migrate` or `collectstatic`, and a guard that blocked those
+would make the mode useless — you would have to turn it off to do the work you turned it on
+for. Only the domain jobs (which write club data, archive clubs, or import members) stand
+down.
+"""
+
+from django.core.management.base import BaseCommand, CommandError
+
+from features.models import Maintenance
+
+
+class MaintenanceAwareCommand(BaseCommand):
+    """A command that must not run while the platform is closed."""
+
+    def execute(self, *args, **options):
+        if Maintenance.is_on() and not options.get("ignore_maintenance"):
+            raise CommandError("The platform is in maintenance mode; this command stands down. Pass --ignore-maintenance to override.")
+
+        return super().execute(*args, **options)
+
+    def create_parser(self, prog_name, subcommand, **kwargs):
+        parser = super().create_parser(prog_name, subcommand, **kwargs)
+        parser.add_argument(
+            "--ignore-maintenance",
+            action="store_true",
+            help="Run even though the platform is in maintenance mode.",
+        )
+        return parser

features/context_processors.py

@@ -0,0 +1,7 @@
+from features.models import Maintenance
+
+
+def maintenance(request):
+    """So no control-panel page can forget the platform is closed. Cached, so it costs no
+    query."""
+    return {"maintenance_on": Maintenance.is_on()}

features/middleware.py

@@ -0,0 +1,74 @@
+"""Platform lock-down.
+
+While maintenance is on, every club subdomain is closed and the base domain keeps only what
+is needed to *end* the maintenance: the control panel, the auth screens that get you into it,
+the static files those pages need, and the health check.
+
+The exemptions are the whole design. Close /accounts/ as well and you cannot sign in to turn
+maintenance off — a lock-down with no key, fixable only from a shell. Close /healthz and the
+load balancer concludes the node is dead and stops routing to it, which takes the control
+panel down with everything else.
+"""
+
+from django.http import JsonResponse
+from django.shortcuts import render
+from django.utils.translation import gettext_lazy as _
+
+from features.models import Maintenance
+
+#: Reachable on the base domain while the platform is locked down.
+OPEN_PREFIXES = (
+    "/controlpanel/",  # the point of the exercise
+    "/accounts/",  # ...which you cannot reach without signing in
+    "/admin/",
+    "/static/",
+    "/media/",
+    "/__reload__/",  # dev only; absent outside DEBUG
+)
+
+#: Reachable on every host, always. The health check must answer or the load balancer will
+#: take the node out of rotation and the control panel with it.
+ALWAYS_OPEN = ("/healthz",)
+
+RETRY_AFTER_SECONDS = 3600
+
+
+class MaintenanceMiddleware:
+    """Runs after ClubTenantMiddleware: whether a request is a club's or the platform's is
+    decided by ``request.club``, which the tenant middleware has just resolved."""
+
+    def __init__(self, get_response):
+        self.get_response = get_response
+
+    def __call__(self, request):
+        if not self.is_closed(request):
+            return self.get_response(request)
+
+        maintenance = Maintenance.current()
+        response = self.render(request, maintenance)
+        response["Retry-After"] = RETRY_AFTER_SECONDS
+
+        return response
+
+    def is_closed(self, request) -> bool:
+        if request.path.startswith(ALWAYS_OPEN):
+            return False
+
+        if not Maintenance.is_on():
+            return False
+
+        # A club subdomain is closed outright — no login, no shop, nothing.
+        if getattr(request, "club", None) is not None:
+            return True
+
+        # The base domain keeps the way back in.
+        return not request.path.startswith(OPEN_PREFIXES)
+
+    def render(self, request, maintenance):
+        message = maintenance.message or _("RosterChief is down for maintenance. It will be back shortly.")
+
+        # An API-ish caller gets JSON rather than a page of HTML it cannot read.
+        if request.headers.get("accept", "").startswith("application/json"):
+            return JsonResponse({"status": "maintenance", "detail": str(message)}, status=503)
+
+        return render(request, "maintenance.html", {"message": message, "maintenance": maintenance}, status=503)

features/migrations/0002_maintenance.py

@@ -0,0 +1,33 @@
+# Generated by Django 6.0.6 on 2026-07-14 08:00
+
+import django.db.models.deletion
+import uuid
+from django.conf import settings
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('features', '0001_initial'),
+        migrations.swappable_dependency(settings.AUTH_USER_MODEL),
+    ]
+
+    operations = [
+        migrations.CreateModel(
+            name='Maintenance',
+            fields=[
+                ('created', models.DateTimeField(auto_now_add=True, verbose_name='created')),
+                ('modified', models.DateTimeField(auto_now=True, verbose_name='modified')),
+                ('id', models.UUIDField(default=uuid.uuid4, editable=False, primary_key=True, serialize=False)),
+                ('is_active', models.BooleanField(default=False, verbose_name='active')),
+                ('message', models.TextField(blank=True, help_text='Shown to clubs while the platform is locked down.', verbose_name='message')),
+                ('started_at', models.DateTimeField(blank=True, null=True, verbose_name='started at')),
+                ('started_by', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='maintenance_windows', to=settings.AUTH_USER_MODEL, verbose_name='started by')),
+            ],
+            options={
+                'verbose_name': 'maintenance',
+                'verbose_name_plural': 'maintenance',
+            },
+        ),
+    ]

features/models.py

@@ -10,11 +10,15 @@ Everything waffle already offers (``everyone`` / ``percent`` / ``staff`` /
 ``superusers`` / per-user / per-group) keeps working untouched.
 """
 
+from django.conf import settings
 from django.db import models
+from django.utils import timezone
 from django.utils.translation import gettext_lazy as _
 from waffle.models import CACHE_EMPTY, AbstractUserFlag
 from waffle.utils import get_cache, keyfmt
 
+from rosterchief.base import UUIDModel
+
 #: Cache key template for a flag's club ids, mirroring waffle's own
 #: FLAG_USERS_CACHE_KEY / FLAG_GROUPS_CACHE_KEY.
 FLAG_CLUBS_CACHE_KEY = "flag:%s:clubs"
@@ -69,3 +73,74 @@ class Flag(AbstractUserFlag):
         if self.everyone is not None:
             return self.everyone
         return club.pk in self._get_club_ids()
+
+
+class Maintenance(UUIDModel):
+    """Platform lock-down. One row, read on every request.
+
+    Cached rather than queried per request, and the cache is the same shared Redis the flags
+    use — so turning maintenance on in the control panel takes effect on every worker and
+    every server at once. A per-process cache would leave some workers still serving clubs.
+    """
+
+    CACHE_KEY = "maintenance:current"
+
+    #: Cached, but not for ever. Write-through makes the flip instant for the process that
+    #: made it and — on the shared Redis of a real deployment — for every other one too. The
+    #: TTL is the belt to that braces: on a per-process cache (a dev box with no Redis, or a
+    #: misconfigured deploy) a lock-down that only reached one gunicorn worker would be worse
+    #: than useless, so the others notice within ten seconds regardless.
+    CACHE_SECONDS = 10
+
+    is_active = models.BooleanField(_("active"), default=False)
+    message = models.TextField(_("message"), blank=True, help_text=_("Shown to clubs while the platform is locked down."))
+    started_at = models.DateTimeField(_("started at"), null=True, blank=True)
+    started_by = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.SET_NULL, null=True, blank=True, related_name="maintenance_windows", verbose_name=_("started by"))
+
+    class Meta:
+        verbose_name = _("maintenance")
+        verbose_name_plural = _("maintenance")
+
+    def __str__(self):
+        return "Maintenance on" if self.is_active else "Maintenance off"
+
+    def save(self, *args, **kwargs):
+        super().save(*args, **kwargs)
+        get_cache().set(self.CACHE_KEY, self, self.CACHE_SECONDS)
+
+    @classmethod
+    def current(cls) -> Maintenance:
+        """The one row, created on first read. Cached until it changes."""
+        cached = get_cache().get(cls.CACHE_KEY)
+        if cached is not None:
+            return cached
+
+        maintenance = cls.objects.first() or cls.objects.create()
+        get_cache().set(cls.CACHE_KEY, maintenance, cls.CACHE_SECONDS)
+
+        return maintenance
+
+    @classmethod
+    def is_on(cls) -> bool:
+        return cls.current().is_active
+
+    @classmethod
+    def start(cls, *, message: str = "", user=None) -> Maintenance:
+        maintenance = cls.current()
+        maintenance.is_active = True
+        maintenance.message = message
+        maintenance.started_at = timezone.now()
+        maintenance.started_by = user
+        maintenance.save()
+
+        return maintenance
+
+    @classmethod
+    def stop(cls) -> Maintenance:
+        maintenance = cls.current()
+        maintenance.is_active = False
+        maintenance.started_at = None
+        maintenance.started_by = None
+        maintenance.save()
+
+        return maintenance

features/tests.py

@@ -1,10 +1,19 @@
+from io import StringIO
+
+from allauth.mfa.models import Authenticator
+from django.contrib.auth import get_user_model
 from django.core.cache import cache
-from django.test import RequestFactory, TestCase
+from django.core.management import call_command
+from django.core.management.base import CommandError
+from django.test import RequestFactory, TestCase, override_settings
 from waffle import flag_is_active, get_waffle_flag_model
 
 from club.models import Club
 
+from .models import Maintenance
+
 Flag = get_waffle_flag_model()
+User = get_user_model()
 
 
 class ClubScopedFlagTests(TestCase):
@@ -94,3 +103,134 @@ class ClubScopedFlagTests(TestCase):
         self.club.flags.add(self.flag)
 
         self.assertTrue(self.active_for(self.club))
+
+
+@override_settings(
+    ROSTERCHIEF_BASE_DOMAIN="rosterchief.app",
+    ALLOWED_HOSTS=["rosterchief.app", "ajax-united.rosterchief.app", "testserver"],
+)
+class MaintenanceModeTests(TestCase):
+    def setUp(self):
+        cache.clear()
+        self.addCleanup(cache.clear)
+        self.club = Club.objects.create(name="Ajax United", slug="ajax-united")
+        self.user = User.objects.create_user(email="root@example.com", password="pw-secret-123", is_staff=True)
+        Authenticator.objects.create(user=self.user, type=Authenticator.Type.TOTP, data={"secret": "JBSWY3DPEHPK3PXP"})
+
+    def club_get(self, path="/"):
+        return self.client.get(path, HTTP_HOST="ajax-united.rosterchief.app")
+
+    def platform_get(self, path):
+        return self.client.get(path, HTTP_HOST="rosterchief.app")
+
+    def test_a_club_subdomain_is_closed(self):
+        Maintenance.start(message="Upgrading the database.")
+
+        response = self.club_get("/accounts/login/")
+
+        self.assertEqual(response.status_code, 503)
+        self.assertContains(response, "Upgrading the database", status_code=503)
+        self.assertEqual(response["Retry-After"], "3600")
+
+    def test_a_club_is_open_again_when_maintenance_ends(self):
+        Maintenance.start()
+        Maintenance.stop()
+
+        self.assertEqual(self.club_get("/accounts/login/").status_code, 200)
+
+    def test_the_control_panel_stays_reachable(self):
+        Maintenance.start()
+        self.client.force_login(self.user)
+
+        self.assertEqual(self.platform_get("/controlpanel/").status_code, 200)
+
+    def test_signing_in_stays_possible(self):
+        # Close /accounts/ as well and you cannot sign in to turn maintenance off: a
+        # lock-down with no key, fixable only from a shell.
+        Maintenance.start()
+
+        self.assertEqual(self.platform_get("/accounts/login/").status_code, 200)
+
+    def test_the_health_check_still_answers(self):
+        # Close it and the load balancer decides the node is dead and stops routing to it —
+        # taking the control panel down with everything else.
+        Maintenance.start()
+
+        self.assertEqual(self.platform_get("/healthz").status_code, 200)
+        self.assertEqual(self.client.get("/healthz", HTTP_HOST="ajax-united.rosterchief.app").status_code, 200)
+
+    def test_the_rest_of_the_base_domain_is_closed(self):
+        Maintenance.start()
+
+        self.assertEqual(self.platform_get("/").status_code, 503)
+
+    def test_a_json_caller_gets_json(self):
+        Maintenance.start(message="Back soon.")
+
+        response = self.client.get("/", HTTP_HOST="ajax-united.rosterchief.app", headers={"accept": "application/json"})
+
+        self.assertEqual(response.status_code, 503)
+        self.assertEqual(response.json()["status"], "maintenance")
+
+    def test_the_message_is_optional(self):
+        Maintenance.start()
+
+        self.assertContains(self.club_get("/"), "down for maintenance", status_code=503)
+
+    def test_it_says_which_state_it_is_in(self):
+        self.assertEqual(str(Maintenance.start()), "Maintenance on")
+        self.assertEqual(str(Maintenance.stop()), "Maintenance off")
+
+    def test_it_records_who_closed_the_platform_and_when(self):
+        maintenance = Maintenance.start(message="db upgrade", user=self.user)
+
+        self.assertTrue(maintenance.is_active)
+        self.assertEqual(maintenance.started_by, self.user)
+        self.assertIsNotNone(maintenance.started_at)
+
+    def test_the_state_is_shared_rather_than_per_process(self):
+        # The cache is the shared one the flags use, so closing the platform reaches every
+        # worker and every server. A per-process cache would leave some workers serving clubs.
+        Maintenance.start()
+
+        self.assertTrue(cache.get(Maintenance.CACHE_KEY).is_active)
+        self.assertTrue(Maintenance.is_on())
+
+        Maintenance.stop()
+
+        self.assertFalse(cache.get(Maintenance.CACHE_KEY).is_active)
+
+
+class MaintenanceCommandTests(TestCase):
+    def setUp(self):
+        cache.clear()
+        self.addCleanup(cache.clear)
+
+    def run_command(self, name, *args):
+        out = StringIO()
+        call_command(name, *args, stdout=out, stderr=out)
+        return out.getvalue()
+
+    def test_a_scheduled_job_stands_down(self):
+        Maintenance.start()
+
+        with self.assertRaises(CommandError):
+            self.run_command("archive_overdue_clubs")
+
+    def test_it_runs_again_once_the_platform_reopens(self):
+        Maintenance.start()
+        Maintenance.stop()
+
+        self.assertIn("Nothing overdue", self.run_command("archive_overdue_clubs"))
+
+    def test_an_override_exists_for_when_you_mean_it(self):
+        Maintenance.start()
+
+        self.assertIn("Nothing overdue", self.run_command("archive_overdue_clubs", "--ignore-maintenance"))
+
+    def test_migrate_is_not_blocked(self):
+        # Maintenance is usually declared IN ORDER to migrate. A guard on every command would
+        # mean turning the mode off to do the work you turned it on for.
+        Maintenance.start()
+
+        self.run_command("migrate", "--check")  # raises SystemExit only if migrations are pending