Operations runbook

Operational procedures for running Wellness Passport in production.

This file starts with the sections US-44 and US-45 own. Deploy, rollback, backup/restore and incident procedures belong here too, and are deliberately not stubbed out below: an empty heading would let the story that owes each of those sections pass its own scenario against a placeholder. Each story adds its section when it lands.

Continuous integration

PR gate vs. nightly (PASS-399 / INF-E3)

bitbucket-pipelines.yml's pull-requests: '**' gate (*gate) is the only step Bitbucket runs automatically, on every PR and every push to main. As of PASS-399 it includes make test-layout — the mobile-first phone-viewport scenarios (PASS-274, -272, -353, -394's 4th scenario each) — alongside lint/typecheck/test-unit/test-api/security-deps. It carries no docker info guard: a runner that loses its docker daemon reds the PR gate rather than silently skipping the mobile-first check CLAUDE.md requires of every UI change. It unsets DOCKER_HOST first, the same as every other docker-touching step in the file — the runner injects DOCKER_HOST=tcp://localhost:2375 for a DinD sidecar, and the tier's ephemeral compose stack publishes its port on the host daemon instead.

make test-layout's ephemeral compose stack (backend/tests/compose_stack.py) runs docker compose up -d --build, which builds docker/frontend.Dockerfile and docker/api.Dockerfile (docker-compose.yml). So the PR gate is now coupled to those two images building correctly — a broken frontend or api image build reds the gate, not just the nightly run.

The heavier tiers — make test-e2e's full PWA/offline/service-worker browser suite, and make test-infra (which also builds the docs image via scripts/build-images.sh) — still run only under custom: nightly. Per the comment above that block in bitbucket-pipelines.yml, Bitbucket never triggers a custom: pipeline on its own; it needs a schedule attached in Repository settings -> Pipelines -> Schedules, which is state outside this repository and outside anything a checked-in test can observe. No such schedule exists yet.

Owner: Anthony Graca (repo admin) is responsible for attaching the nightly schedule and for triaging a red nightly run.
Policy if nightly goes red: treat it as a same-day blocker on the next deploy, not a queued backlog item — freeze further deploys (DEPLOY_UNFROZEN, see scripts/deploy-gated.sh) until the regression is understood. A red nightly means a container image, a k8s manifest, or the non-mobile browser suite is broken on main, and — until the schedule above exists — nothing else in this pipeline will catch it before it reaches production.
Until the schedule is attached, those tiers are only actually exercised when someone triggers custom: nightly by hand. backend/tests/tier_deferral_conventions.py's TIER_DEFERRALS registry (PASS-348) currently treats make test-infra/make test-e2e appearing anywhere in bitbucket-pipelines.yml — including under custom: — as proof of enforcement, which is more generous than the schedule gap above actually earns. PASS-399 added the sharper distinction (automatically_triggered_make_targets() in backend/tests/pipeline_conventions.py, which excludes custom:) but did not rewire the existing tooling/browser TierDeferral entries onto it — doing so would red this repo's own gate over a schedule this repo cannot create for itself. Tightening that check, or attaching the schedule, is tracked as follow-up.

Gate-duration review (PASS-416 / INF-E3)

PASS-399 coupled the PR gate to two docker image builds (docker/frontend.Dockerfile, docker/api.Dockerfile) and a Chromium install (playwright install chromium) on the single self-hosted runner where concurrent PRs serialize — accepted deliberately, but new gate surface. That story measured make test-layout locally at 19.2s wall clock with images already built (well inside its own ~10-15s compose-startup sizing) but had no per-command visibility into the gate's own duration, and no declared cache for the Playwright browser binary — an unstated assumption that the runner's persistent ~/.cache/ms-playwright survives between runs.

PASS-416 ships the instrument: scripts/gate-timing.sh, sourced into &gate and wrapping every command that can fail with gate_time, plus a gate_report in after-script that prints every command's real elapsed time, slowest first — so the review below has actual data instead of a single Bitbucket-reported step duration. It also declares the missing ms-playwright cache (~/.cache/ms-playwright, matching Playwright's own default browsers directory), closing the one residual that didn't need two weeks of data to justify. What is not shipped: a docker layer-cache strategy for the two images — that stays deferred pending what the review below finds, per the acceptance criteria.

Gate-duration review owner: Anthony Graca (repo admin).
Gate-duration review window: 2026-07-25 through 2026-08-08 — read gate_report's output from the gate's own step log across PRs merged in that window before deciding whether to act on either residual below.
Gate-duration adoption threshold: if the window's make test-layout runs have a median above 90s, or make install-e2e runs have a median above 30s, that means image-rebuild or browser-download cost is dominating rather than the ~19.2s measured with a warm cache — add a docker layer-cache strategy for docker/frontend.Dockerfile / docker/api.Dockerfile and re-verify the ms-playwright cache is actually hitting (a runner rebuild or a Playwright version bump can still force a re-download despite the cache being declared). Below both thresholds, no action is needed — the declared caches and the current image-build cost are doing their job.

Deployment

US-38 / PR-D1 — the four scenarios in backend/tests/features/staging_deployment.feature, asserted by backend/tests/test_staging_deployment_bdd.py. Container builds and production's own deploy mechanics (commit-tagged images, the k8s/ kustomize base, the Bitbucket Pipelines gate) were already covered by US-48/US-49/ US-51 (INF-C1, INF-D1/D2, INF-E3) before this story landed; what this section adds is the staging tier those stories' own documentation flagged as missing (docs/diagrams/deployment/pass53_dast_scan_topology.puml, docs/security-dast-baseline.md's "No staging tier" residual gap) and the rollback procedure this file had, until now, only reserved a promise for.

Standing up staging

Staging is not a hand-maintained second copy of production's manifests — it is k8s-staging/kustomization.yaml, a kustomize overlay whose only resources: entry is the production base (../k8s). Every Deployment, Service, probe, and container port staging runs is therefore byte-identical to production's; the overlay is only permitted to patch three things:

Namespace — wellness-passport-staging, vs. production's wellness-passport.
Ingress hostnames — staging-demo.greaterangelssoftware.com and staging-docs.greaterangelssoftware.com, vs. production's demo.greaterangelssoftware.com / docs.greaterangelssoftware.com.
Image tag — staging carries its own images: block in k8s-staging/kustomization.yaml, re-pinned independently of production's, so the two environments can run different commits.

A second engineer stands up staging with the identical entry point that deploys production, scripts/deploy.sh, pointed at the overlay:

K8S_DIR=k8s-staging NAMESPACE=wellness-passport-staging \
  DEMO_HOST=https://staging-demo.greaterangelssoftware.com \
  DOCS_HOST=https://staging-docs.greaterangelssoftware.com \
  scripts/deploy.sh

This runs the same build → push → kubectl apply -k k8s-staging/ → wait for the migration Job → wait for rollouts → smoke-test → verify-publication pipeline production's own deploy uses (see the Bitbucket Pipelines' gated merge step), just against the staging namespace and hostnames. scripts/smoke.sh's DEMO_HOST/DOCS_HOST overrides and scripts/verify-publication.sh's matching overrides are what let the same scripts assert reachability against either environment's real URL.

The one one-time, human-run prerequisite (mirroring bootstrap/'s role for production) is that the wellness-passport-staging namespace needs its own gitea-registry image-pull secret before the first deploy, created out-of-band exactly as production's is — kubectl create secret docker-registry gitea-registry -n wellness-passport-staging ... — since that credential is deliberately never committed to the repository (see docs/secrets-management's local-secret-file convention).

Production vs. staging — the enumerated configuration differences

Per the structure above, staging differs from production in exactly these configuration values, and nothing else:

Axis	Production	Staging
Namespace	`wellness-passport`	`wellness-passport-staging`
Demo/API hostname	`demo.greaterangelssoftware.com`	`staging-demo.greaterangelssoftware.com`
Docs hostname	`docs.greaterangelssoftware.com`	`staging-docs.greaterangelssoftware.com`
Image tag	whatever the last production deploy pinned	whatever the last staging deploy pinned — independently

Every other manifest field — replica counts, probes, container ports, env var names, Secret keys, resource requests — comes from the one shared base (k8s/) both environments compose, so it cannot drift between them without also changing production. backend/tests/test_staging_deployment_bdd.py enforces this structurally on every make test-api run, by rendering both and diffing every Deployment outside these four axes.

Which commit is a pod running?

Every pod a deploy rebuilds carries the deployed tag as app.kubernetes.io/version, so the question is a label query rather than string-surgery on an image reference:

kubectl get pods -n wellness-passport -L app.kubernetes.io/version

NAME                        READY   STATUS    AGE   VERSION
api-79d9cbf8c4-xwq4f        1/1     Running   89m   0.1.0-3e64ac527ae1
docs-7bd54df4dc-8dbdx       1/1     Running   89m   0.1.0-3e64ac527ae1
frontend-5cbcbdf6fd-nwxqw   1/1     Running   89m   0.1.0-3e64ac527ae1
postgres-cd6979f56-d5brb    1/1     Running   26h

The tag is <VERSION>-<sha12>, so the trailing twelve characters are a commit: git log -1 3e64ac527ae1 resolves it in this repo. Because it is a real label it also selects, which is what makes it useful during an incident — find every pod still on a superseded build, across namespaces:

kubectl get pods -A -l app.kubernetes.io/version=0.1.0-fddb4b3d6f86

postgres is deliberately unlabelled and shows blank above. It is not built from this repo, and labelling it would put a changing value in its pod template — which, since it is emptyDir-backed, would make every deploy restart it and drop the database. k8s/version-stamp.yaml records that reasoning; scripts/deploy.sh rewrites the tag into it on each deploy, the same way it re-pins kustomization.yaml's newTag.

To ask a running process what it is — rather than what its spec claims, which differs if a tag was overwritten or a stale layer was served — the api serves its own stamp:

kubectl exec -n wellness-passport deploy/api -- \
  python -c "import urllib.request,sys; sys.stdout.write(urllib.request.urlopen('http://localhost:8000/api/version').read().decode())"

{"commit":"3e64ac527ae1d0f9...","version":"0.1.0","build_time":"2026-07-22T11:02:04-0700"}

It is public through the ingress too (https://demo.greaterangelssoftware.com/api/version), which is the quickest way to confirm what the edge is actually serving after a deploy. The SPA carries the identical three values — scripts/build-images.sh passes one commit/version/time triple to both images — so api and frontend disagreeing about their own provenance means something shipped out of band.

The namespace-level wellness-passport.io/last-deployed-tag annotation records what the last deploy pinned. That is the right input for rollback (below) and the wrong one for this question: it is a single value for the whole namespace, so it cannot describe a namespace mid-rollout or one still holding pods from an earlier release.

Rollback

scripts/rollback.sh restores the release that was running immediately before the current one:

scripts/rollback.sh                                   # production, one step back
K8S_DIR=k8s-staging NAMESPACE=wellness-passport-staging \
  scripts/rollback.sh                                  # staging, one step back
TARGET_TAG=abc1234 scripts/rollback.sh                 # a specific known-good tag

It reads the target namespace's wellness-passport.io/last-deployed-tag annotation — the same one scripts/deploy.sh stamps after every successful deploy — to find the currently-running tag, then walks the api Deployment's ReplicaSet history to the newest one that is not that tag, and redeploys exactly that tag through scripts/deploy.sh with SKIP_BUILD=1. Rollback never rebuilds: it always redeploys bits the registry already has and this pipeline already verified once, so a rollback cannot itself fail for a reason unrelated to the release being reverted.

Drilling rollback

Rehearsed against staging (never production) before this section is treated as load-bearing. This is now a runnable drill, scripts/staging-rollback-drill.sh (PASS-275), not a prose checklist — it performs exactly these steps and exits non-zero, naming what the staging URL is actually answering with, if the release does not revert:

Deploy a known tag to staging: K8S_DIR=k8s-staging NAMESPACE=wellness-passport-staging TAG=<tag-a> SKIP_BUILD=1 DEMO_HOST=https://staging-demo.greaterangelssoftware.com DOCS_HOST=https://staging-docs.greaterangelssoftware.com scripts/deploy.sh.
Deploy a second tag on top of it: ... TAG=<tag-b> SKIP_BUILD=1 scripts/deploy.sh. Confirm staging-demo.greaterangelssoftware.com answers with <tag-b> — the drill fetches the site and reads the provenance stamp it serves, and scripts/verify-publication.sh asserts the same during the deploy. This is an HTTP observation of what the URL answers, not a read of the api Deployment's declared pod-template image.
Run K8S_DIR=k8s-staging NAMESPACE=wellness-passport-staging scripts/rollback.sh.
Confirm the rollout completes and staging-demo.greaterangelssoftware.com answers with <tag-a> again — the previous release, restored, observed at the URL.

Run it two ways:

Every PR, no cluster needed. backend/tests/test_staging_rollback_drill_bdd.py runs the drill against the fake-cluster tool doubles in backend/tests/fake_cluster/: the real scripts/deploy.sh and scripts/rollback.sh bytes execute — only kubectl/kustomize/curl are doubled, and the curl double answers with the release the cluster state says is served — so a rollback that redeploys the wrong tag leaves the staging URL answering with the wrong release and turns the drill red, the gap US-38's script-text greps left open. What it does not prove is that a real Kubernetes API server behaves as the doubles model (ReplicaSet garbage collection, --sort-by semantics, jsonpath escaping); that residual is the live drill's job.
Against the real staging cluster. WP_STAGING_DRILL=1 make test-cluster (or scripts/staging-rollback-drill.sh by hand) drives the identical script against whatever cluster kubectl points at. Prerequisite: the drill deploys and reverts already-built tags — it never builds (SKIP_BUILD=1) — so you must first push two real staging releases and name them, along with the provenance commit each image carries (what the staging URL will answer with): TAG_A=<tag> STAMP_A=<commit> TAG_B=<tag> STAMP_B=<commit> WP_STAGING_DRILL=1 make test-cluster. Without them the drill fails loud rather than deploying an unpushed placeholder that would ImagePullBackOff. The pytest scenario is gated: it skips cleanly when WP_STAGING_DRILL is unset, and fails loud — never skips — if the flag is set but no cluster answers (or the required tags are missing).

Backup and restore

US-34 / PR-B3 — the three scenarios in backend/tests/features/backups_verified_restore.feature, asserted by backend/tests/test_backups_verified_restore_bdd.py. A backup that has never been restored is a hope, not a capability — this section exists so a launch does not ship on the strength of an unexercised script.

Cadence

k8s/backup-cronjob.yaml runs daily (0 3 * * *, UTC) against the production database, writing into the backup-storage PersistentVolumeClaim (k8s/backup-pvc.yaml). backend/tests/backup_restore_conventions.py's max_gap_between_fires() parses that CronJob's own spec.schedule and computes the actual maximum gap between fires — a check against the deployed manifest, not a label trusted at face value. The PR gate enforces that computed gap never exceeds 24 hours.

Every run — success or failure — writes a *.manifest.json recording the outcome, start/finish timestamps, the dump filename, the table count captured, and (on failure) a populated reason. That manifest is what makes a failed backup visible to the operator: a monitoring job (or a human) reads it rather than trusting an unwatched exit code. scripts/backup-db.sh is the same procedure, runnable by hand from an operator's workstation (which has no in-cluster pg_dump but does have Docker) against any WP_DATABASE_URL.

Residual gaps, recorded rather than hidden:

backup-storage assumes a default StorageClass is provisioned on the cluster — an environment with none configured will fail to bind the PVC.
The PVC lives in the same cluster as the database it protects. A cluster- or node-level disaster takes both down together; an off-cluster backup destination (e.g. object storage) is not yet wired up.

Recovery objectives

Maximum acceptable data loss window (RPO): 24 hours — matches the daily backup cadence above; a failure between two runs can lose at most one day's check-ins. Expected time-to-restore (RTO): 60 minutes — the time budget for provisioning a scratch target, running scripts/restore-db.sh, and confirming the integrity checks pass, based on the restore drill below. Neither figure has yet been ratified by SHS; both are recorded as the numbers CI enforces (backend/tests/backup_restore_conventions.py's RPO / RTO) so an agreed figure, when it arrives, replaces a stated number rather than a blank — the same posture the Load-testing section above already takes on its own unratified targets.

Restore drill

The documented restore procedure: run scripts/backup-db.sh against the source database, then scripts/restore-db.sh <dump-file> <target-url> against a scratch PostgreSQL target. restore-db.sh verifies pg_restore exited 0 and that the restored database's table count matches the dump's own table-of-contents count. The stronger claim — that the seeded rows survive the round trip, not merely that the tables exist — is asserted by querying the restored database's row values and counts directly; a pg_restore --schema-only run (which recovers every table name with zero rows) fails that assertion, which is the gap this drill exists to close.

Drill log

Date	Outcome	Notes
2026-07-25	Success	`make test-infra`'s restore-drill scenario (`test_backups_verified_restore_bdd.py::test_restore_drill_is_executed_before_launch`) run against two ephemeral `postgres:16-alpine` containers. `backup-db.sh` dumped a seeded source (1 student, 1 challenge, 1 task, 1 check-in); `restore-db.sh` restored it into a scratch target and passed its table-count check; the scenario's own row-level assertions confirmed all four seeded rows survived by value.

Security operations

US-44 / PR-F2 — the four scenarios in backend/tests/features/vendor_security_package.feature, asserted by backend/tests/test_vendor_security_package_bdd.py. The vendor security package itself — docs/hecvat-lite.md and docs/security-dast-baseline.md — lives alongside this runbook, not inside it; this section records the ongoing cadence, not the one-time assessment.

Dependency-audit gate

make security-deps runs on every change, in both CI systems (.github/workflows/ci.yml, bitbucket-pipelines.yml):

npm audit --omit=dev --audit-level=high — shipped frontend dependencies, failing the build at High or Critical.
pip-audit — every backend dependency, failing the build on any reported advisory (its own default posture, stricter than npm's floor).

A finding that cannot be fixed immediately (no upstream release, or the fix is a breaking major-version bump tracked as its own upgrade) is recorded in the accepted-risk register below with a rationale — never silently suppressed with an unrecorded --ignore-vuln or an unreviewed lockfile pin. As of this writing both audits are clean; the register has no entries.

DAST cadence

An OWASP ZAP baseline scan runs against the local docker-compose.yml stack (make security-dast) — see docs/security-dast-baseline.md for the procedure, the no-staging-tier caveat, and the full triage. Re-run:

before every production-bound release that touches routing, headers, middleware, or a new externally-reachable endpoint;
quarterly otherwise, so a finding introduced by a dependency bump (not a code change) doesn't go unnoticed between releases.

Authorization sweep

backend/tests/test_vendor_security_package_bdd.py's Scenario 4 runs on every make test-api — it is not a separate cadence, it is the PR gate. The route set it sweeps is derived from each route's dependency tree (backend/tests/vendor_security_conventions.py), not a maintained prefix list, so a new router under any path is swept automatically; an unguarded route fails the build the same as any other test failure.

Push subscription storage (PASS-387 / FR-C7, US-67)

WP_PUSH_ENABLED (default false) ships the feature dark; turning it on in production requires WP_VAPID_PUBLIC_KEY / WP_VAPID_PRIVATE_KEY to both be set — the app refuses to boot otherwise (app/config.py's production validator). Rotating the VAPID public key invalidates every existing subscription: a browser's stored PushSubscription is bound to the public key it was created against, so once the key changes, every previously registered device silently stops receiving pushes until it re-subscribes. Treat a VAPID rotation as a re-subscribe event, not a silent secret swap — plan for a period where old subscriptions are dead rows (they are pruned naturally the next time each device re-registers; push_subscriptions.endpoint is the upsert key, see app/services/notifications.py) rather than expecting an in-place credential swap with no user-visible effect.

Accepted-risk register

Every accepted (not resolved) dependency-audit or DAST finding is recorded with its rationale at the point it was triaged — docs/hecvat-lite.md's VULN-1 row for a dependency-audit exception, or the per-finding table in docs/security-dast-baseline.md for a DAST finding — rather than duplicated here. As of this writing: no dependency-audit exceptions exist (VULN-1); all 15 DAST findings are Accepted with rationale (docs/security-dast-baseline.md), none above Medium risk.

Load testing

US-45 / PR-F3 — the three scenarios in backend/tests/features/event_scale_load_validation.feature, asserted by backend/tests/test_event_scale_load_validation_bdd.py. That module checks this section records a method, a scale, and results; that the targets published here are the same constants the scenarios assert against (backend/tests/load_validation_conventions.py); and that every result recorded here met its own target.

Date: 2026-07-20 · Build: feature/PASS-54-us-45-event-scale-load-validation

Method

An in-process load harness, run as part of the ordinary pytest suite rather than as a separate tier:

Cohort. 1,000 students and their enrollments are bulk-inserted straight into the database (seed_enrolled_students). The cohort is the fixture, not the subject — routing 2,000 HTTP requests to reach the starting line would add minutes and measure nothing under test.
Burst. 300 students sign in and each POSTs /api/checkins/scan with a freshly minted event token, driven sequentially through Starlette's TestClient. Only the scan itself is timed; sign-in is excluded, because a real event burst is students who are already signed in scanning a poster.
Report. A semester of check-in data is seeded funnel-shaped (everyone completes week 1, each later week retains ~85% of the one before) and GET /api/reports/participation is timed once, end to end, with small-cell suppression in force — privacy is on in production, so it is on in the measurement.
Percentile. Nearest-rank p95 (p95()), so the reported figure is a latency that was actually observed rather than an interpolation between two that were.

Reproduce with:

cd backend && python -m pytest tests/test_event_scale_load_validation_bdd.py \
    --log-cli-level=INFO -k "burst or responsive"

The load: lines in that output are the figures tabulated below.

Scale

Dimension	Figure	Source
Enrolled students	1,000	US-45 Description — fall scale
Check-ins in the burst	300	US-45 scenario 1 — "hundreds in a few minutes"
Burst arrival window	5 minutes	US-45 scenario 1
Check-in rows behind the report	4,523	1,000 students × 6 weeks, 85% weekly retention

Results

Three consecutive runs on the CI-class developer machine, 2026-07-20. The observed column is the worst figure across those runs, not the best.

Measurement	Target	Observed	Verdict
Check-in p95	400 ms	15.8 ms	Pass — 25× headroom
Participation report	2,000 ms	8.7 ms	Pass — 230× headroom

Supporting figures from the same runs: check-in p50 13.6–13.8 ms, max 72.6–76.2 ms (the max is the first request of the burst, which pays one-time import and connection setup). All 300 check-ins returned 200; none failed, and none was throttled. The 300-request burst completed in 7.2–7.5 s of wall clock, well inside the 5-minute arrival window it models.

Where the targets come from. 400 ms for a check-in is an interaction budget: the student is standing at a poster with the phone camera open, and the response should land before they lower it. 2,000 ms for the participation report is a dashboard-refresh budget for an aggregate over the whole cohort. Both are proposed here by the delivery engineer and have not yet been ratified by the SHS stakeholders — see the residual gap below.

Residual gaps

Recorded here rather than left implied, in the posture docs/a11y-manual-pass.md takes on its screen-reader gap.

This is not a production measurement. The harness runs in-process against a single-writer in-memory SQLite with no network, no Postgres, no concurrency, and no ingress. What it validates is that the application's own work does not grow badly with the size of the cohort — the failure this story exists to prevent. It does not tell you what the box will do on event night. A run against the compose stack (make up) or the k8s demo topology, with concurrent clients, is outstanding.
The targets are proposed, not agreed. Neither 400 ms nor 2,000 ms has been signed off by SHS. They are recorded as the numbers CI enforces so that the agreed figures, when they arrive, replace a stated number rather than a blank.
The rate limiter is per-process and in-memory (backend/app/rate_limit.py). Under N replicas the effective ceiling is N × the configured limit, and a student's burst is only throttled consistently if their requests land on one replica. At the current single-replica demo topology this is not reachable; it becomes real the moment the API is scaled out. See docs/diagrams/deployment/pass54_load_topology.puml.
Concurrency is untested. The burst is sequential, so nothing here exercises lock contention, connection-pool exhaustion, or the uq_checkin_student_task race under simultaneous duplicate scans. The duplicate race is covered functionally by US-8's suite, but not under load.