Operations runbook

Operational procedures for running Wellness Passport in production.

This file starts with the sections US-44 and US-45 own. Deploy, rollback, backup/restore and incident procedures belong here too, and are deliberately not stubbed out below: an empty heading would let the story that owes each of those sections pass its own scenario against a placeholder. Each story adds its section when it lands.

Continuous integration

PR gate vs. nightly (PASS-399 / INF-E3)

bitbucket-pipelines.yml's pull-requests: '**' gate (*gate) is the only step Bitbucket runs automatically, on every PR and every push to main. As of PASS-399 it includes make test-layout — the mobile-first phone-viewport scenarios (PASS-274, -272, -353, -394's 4th scenario each) — alongside lint/typecheck/test-unit/test-api/security-deps. It carries no docker info guard: a runner that loses its docker daemon reds the PR gate rather than silently skipping the mobile-first check CLAUDE.md requires of every UI change. It unsets DOCKER_HOST first, the same as every other docker-touching step in the file — the runner injects DOCKER_HOST=tcp://localhost:2375 for a DinD sidecar, and the tier's ephemeral compose stack publishes its port on the host daemon instead.

make test-layout's ephemeral compose stack (backend/tests/compose_stack.py) runs docker compose up -d --build, which builds docker/frontend.Dockerfile and docker/api.Dockerfile (docker-compose.yml). So the PR gate is now coupled to those two images building correctly — a broken frontend or api image build reds the gate, not just the nightly run.

The heavier tiers — make test-e2e's full PWA/offline/service-worker browser suite, and make test-infra (which also builds the docs image via scripts/build-images.sh) — still run only under custom: nightly. Per the comment above that block in bitbucket-pipelines.yml, Bitbucket never triggers a custom: pipeline on its own; it needs a schedule attached in Repository settings -> Pipelines -> Schedules, which is state outside this repository and outside anything a checked-in test can observe. No such schedule exists yet.

Owner: Anthony Graca (repo admin) is responsible for attaching the nightly schedule and for triaging a red nightly run.
Policy if nightly goes red: treat it as a same-day blocker on the next deploy, not a queued backlog item — freeze further deploys (DEPLOY_UNFROZEN, see scripts/deploy-gated.sh) until the regression is understood. A red nightly means a container image, a k8s manifest, or the non-mobile browser suite is broken on main, and — until the schedule above exists — nothing else in this pipeline will catch it before it reaches production.
Until the schedule is attached, those tiers are only actually exercised when someone triggers custom: nightly by hand. backend/tests/tier_deferral_conventions.py's TIER_DEFERRALS registry (PASS-348) currently treats make test-infra/make test-e2e appearing anywhere in bitbucket-pipelines.yml — including under custom: — as proof of enforcement, which is more generous than the schedule gap above actually earns. PASS-399 added the sharper distinction (automatically_triggered_make_targets() in backend/tests/pipeline_conventions.py, which excludes custom:) but did not rewire the existing tooling/browser TierDeferral entries onto it — doing so would red this repo's own gate over a schedule this repo cannot create for itself. Tightening that check, or attaching the schedule, is tracked as follow-up.

Alerting

Nothing pages anyone today. Read this before assuming an outage will find you.

k8s/monitoring/ carries a working Prometheus + Alertmanager pair with two real rules — ApiDown (up{job="api"} == 0, for 2m) and ApiHigh5xxRate (for 5m), both routed to a receiver named oncall. What it has never had is a destination: alertmanager-secret.yaml ships the placeholder https://REPLACE-ME.example.com/oncall-webhook, and kubectl apply -k k8s/monitoring/ accepted it happily. Alerts evaluated, fired, POSTed to a hostname that does not resolve, and every layer reported success.

Two consequences, and the second is the larger one:

Apply monitoring through the wrapper, never kubectl apply -k directly (PASS-702). scripts/apply-monitoring.sh refuses to apply unless ONCALL_WEBHOOK_URL is set to a real https:// destination, and substitutes it at apply time so the URL — which is a bearer credential — never lands in the repository:

ONCALL_WEBHOOK_URL='https://…' scripts/apply-monitoring.sh

A reachable URL is still not a delivered page. Fire one real alert and confirm it arrives before treating alerting as live.

This covers the self-hosted cluster only. ADR 0001 puts production on AWS, where none of it runs: no Prometheus, nothing scraping /metrics, so ApiDown and ApiHigh5xxRate do not exist there at all.

PASS-382 / US-86 closes that with CloudWatch alarms declared in the CDK, every one of them wired to a single SNS topic (Alerts) whose only subscriber is the address passed as -c alertEmail=. Whichever tier is deployed, the same four things are watched: a 5xx RATIO rather than a count (twenty failures is an outage at a hundred requests and a rounding error at a hundred thousand), p95 latency above 3s, the deploy-time migrate path failing, and the reminder scheduler failing silently. infra/test/*.test.ts asserts that every alarm has the topic as an action — an alarm with no action is worse than no alarm, because it reads as coverage.

Two manual steps, and neither is optional. An email subscription stays PENDING and delivers nothing until a human clicks AWS's confirmation link, so confirm it and then fire one real alarm and watch it arrive. And synthesizing without -c alertEmail= emits a synth-time WARNING for exactly this reason: every alarm would fire into a topic nobody is subscribed to.

Log retention is explicit (one year) on every log group the stacks create, rather than the platform default of never expiring.

Deployment

US-38 / PR-D1 — the four scenarios in backend/tests/features/staging_deployment.feature, asserted by backend/tests/test_staging_deployment_bdd.py. Container builds and production's own deploy mechanics (commit-tagged images, the k8s/ kustomize base, the Bitbucket Pipelines gate) were already covered by US-48/US-49/ US-51 (INF-C1, INF-D1/D2, INF-E3) before this story landed; what this section adds is the staging tier those stories' own documentation flagged as missing (docs/security-dast-baseline.md's "No staging tier" residual gap) and the rollback procedure this file had, until now, only reserved a promise for.

Production on AWS (PASS-382 / US-86)

Production is the topology docs/adr/0001-production-hosting-topology.md selected, declared as CDK under infra/. It is a different thing from the self-hosted cluster the rest of this section describes — that cluster is the demo and docs tier, and the staging table below is about IT, not about this.

The compute tier is ECS Fargate behind an ALB, declared in infra/lib/service-stack.ts:

cd infra
export AWS_PROFILE=wp-prod   # or pass --profile wp-prod on every command
npx cdk deploy --all -c imageTag=<version>-<sha12>

What it costs. docs/aws-cost-model.md prices this topology line by line, with every unit price verified against the AWS Pricing API and every traffic volume labelled as an estimate. It also carries the cost levers — the largest is -c endpointAzCount=1, worth ~$350/year.

Feature flags. Three features ship dormant and are turned on per deploy. Each is a PAIR — a backend environment variable and a frontend build argument — and nothing wires the two together at runtime, so they must be set in lockstep. Mismatched, the SPA shows a control whose request fails: with its flag off the backend does not mount the router at all, so the call 404s.

Feature	Backend env var	Frontend build arg	Default
Wellness guide (PASS-378)	`WP_GUIDE_ENABLED`	(not wired — see below)	off
Rehearsal mode (PASS-717)	`WP_REHEARSAL_ENABLED`	`REHEARSAL_ENABLED`	off
Getting started (PASS-718)	`WP_ONBOARDING_ENABLED`	`ONBOARDING_ENABLED`	off

The frontend half is BUILD-time, not run-time: Vite compiles the value into the bundle, and a static asset has no config surface to poll. Turning one on therefore needs a rebuilt and republished SPA, not just an environment change.

There are two frontend build paths and the flags must be passed to whichever one a given tier uses:

# Production (ADR 0001: S3 + CloudFront). This is the shipped SPA.
REHEARSAL_ENABLED=true ONBOARDING_ENABLED=true scripts/publish-frontend.sh

# The container image (compose, and the k8s demo/docs tier).
docker build -f docker/frontend.Dockerfile \
  --build-arg ONBOARDING_ENABLED=true \
  --build-arg REHEARSAL_ENABLED=true .

Production does NOT serve the frontend image. Passing the build args to the Dockerfile alone changes nothing a real user sees.

The dev and browser-tier stack is the exception: docker-compose.yml sets both flags on, for its frontend build args and the api's environment together, so make up and the layout tier exercise the features rather than a build that no longer has them.

Unset means empty, and frontend/src/config/features.ts compares against the string "true", so a plain build with no --build-arg produces an image with every feature dormant. That is the intended default: nothing has to be remembered for a deploy to be safe, only for one to be turned on.

VITE_GUIDE_ENABLED has deliberately never been added to docker/frontend.Dockerfile. The guide has been dormant since PASS-378 and is meant to stay so for the pilot; wiring it would be a behaviour change, not plumbing. Turning the guide on needs that argument added first.

Rehearsal mode has one asymmetry worth knowing before flipping it off again: a session cookie outlives a deploy. An admin who was rehearsing when the flag went off still carries the claim, which routes them into the student app, and /api/rehearsal/exit is no longer mounted to return them. They recover by signing out and back in, which mints a session without it.

Deploy order, first time.

Authenticate first, and then make the profile ambient:

aws login --profile wp-prod
export AWS_PROFILE=wp-prod      # do NOT skip this

aws login --profile wp-prod refreshes credentials for that profile. It does NOT change what default points at, and it does NOT make any later tool use it. The CDK CLI with no --profile falls back to the default profile, which here is the management account 746050097020, and every command below then fails with:

Could not assume role in target account using current credentials (which are for account 746050097020) … is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::743050012859:role/cdk-hnb659fds-deploy-role-…

That error's advice to re-bootstrap with --trust is wrong for this cause. The role exists and is correct; the credentials are simply for the wrong account. Re-bootstrapping would churn working infrastructure and not fix it. Check with aws sts get-caller-identity — it must print 743050012859 — before concluding anything about the bootstrap.

Every cdk command below therefore also carries --profile wp-prod explicitly, so a copied line works even in a shell where the export was forgotten. Belt and braces on purpose: getting this right four times and wrong the fifth would deploy WpEdge or WpFrontend into the management account, which creates real resources in the wrong place rather than erroring.

npx cdk deploy WpNetwork WpRegistry WpData --profile wp-prod
Get an image with a real tag into the ECR repository WpRegistry created (RepositoryUri output). Nothing deploys until one exists — -c imageTag=latest synthesizes with a warning and must never be used for a real deploy, because it makes the running artifact unidentifiable and makes both rollback scripts unable to tell two releases apart.

This step is MANUAL today. There is no CI job that publishes the API image (bitbucket-pipelines.yml only smoke-builds the docs image), so an earlier version of this line saying "push the CI image tag" described a path that does not exist. Tracked as PASS-712. Until that lands:

```bash # the tag CI will eventually produce: - TAG="$(cat VERSION)-$(git rev-parse HEAD | cut -c1-12)" ECR=743050012859.dkr.ecr.us-west-2.amazonaws.com

# PUSH THE COMMIT FIRST. The image bakes APP_COMMIT in and serves it from # GET /api/version; a stamp naming a commit that exists only on one laptop # is the same defect as :latest, in a less obvious costume. git push -u origin HEAD

# Build through the single blessed entry point. PUSH=0 deliberately: the # script builds api, frontend AND docs, and only wellness-passport-api # exists in ECR, so PUSH=1 aborts partway under set -euo pipefail. REGISTRY="$ECR" TAG="$TAG" PUSH=0 scripts/build-images.sh

aws ecr get-login-password --profile wp-prod --region us-west-2 \ | docker login --username AWS --password-stdin "$ECR"

docker push "$ECR/wellness-passport-api:$TAG" ```

Verify before deploying, because a push that half-failed still leaves a tag:

bash aws ecr describe-images --profile wp-prod --region us-west-2 \ --repository-name wellness-passport-api \ --query 'imageDetails[].{Tags:imageTags,Digest:imageDigest,Pushed:imagePushedAt}'

ECR tags here are immutable, so re-pushing an existing tag fails rather than silently repointing it. That is the property both rollback scripts depend on — do not disable it to "fix" a failed push; build a new commit instead.

Immutability cuts both ways, and the second edge is the dangerous one. ECR keeps every tag ever pushed and each one still pulls forever. So a tag computed from a stale main does not error — it resolves, deploys cleanly, and silently rolls production backwards. parseImageTag refuses an absent tag (log 2.14); nothing refuses a stale one, because staleness is invisible from the tag alone. It is only visible by comparing the tag against what is currently serving.

Observed 2026-08-31: local HEAD was 2781887f1004 on a feature branch while origin/main was still b28df2630388, three commits behind. Both tags were in ECR. Deploying the one derived from HEAD on a fresh main would have reverted the fix that lets any student sign in at all — successfully, and without a single error.

Use scripts/deploy-prod.sh rather than assembling this by hand. It resolves the tag from origin/main (never the working copy), refuses a tag ECR does not hold, refuses anything that is not a fast-forward from the running commit, deploys, and then confirms against /api/version — which reports the commit baked into the image, not the tag that was requested.

bash scripts/deploy-prod.sh # deploy origin/main DRY_RUN=1 scripts/deploy-prod.sh # resolve and verify only REF=<commit> scripts/deploy-prod.sh # a specific commit ALLOW_ROLLBACK=1 scripts/deploy-prod.sh # deliberate rollback

The fast-forward check is a hard stop rather than a prompt on purpose: the dangerous case and the safe one are indistinguishable at a glance, since both tags exist, both pull, and both deploy without complaint. 3. (Step retired with the Lambda tier; numbering kept so older notes still line up.)

FIRST DEPLOY ONLY — deploy the Fargate tier at zero tasks first. The sequence below is circular against a database no migration has touched, and the circularity is invisible until it costs a rollback:
step 4 deploys WpService, whose wait-for-migration container runs python -m app.migrate check. On a fresh database app/schema_guard.py refuses — "a genuinely fresh database: refuse, pointing at the deploy-time migrate step". It exits non-zero, the api container's SUCCESS dependency is never satisfiable, every task fails to start, and the ECS deployment circuit breaker rolls the whole stack back;
step 8 runs that migration — using MigrateTaskDefinitionArn, which is an output of the step 4 that just rolled back.

Break the cycle by creating the task definitions without starting a service task:

bash npx cdk deploy WpService --profile wp-prod \ -c imageTag=<tag> -c alertEmail=<address> -c desiredCount=0

Then run step 8's migration now, against the task definition that deploy produced, and come back for step 4 proper. On every LATER deploy the schema is already at head, the gate passes, and -c desiredCount=0 is unnecessary.

4b. npx cdk deploy WpService --profile wp-prod, with -c imageTag= and -c alertEmail=. The deploy WAITS on an ACM certificate: DNS is on Cloudflare, so add the validation CNAME the console shows you, by hand. Pass the issued ARN back as -c apiCertificateArn= on later deploys to skip that wait. 5. npx cdk deploy WpEdge --profile wp-prod — us-east-1, and only because a CloudFront viewer certificate and a CLOUDFRONT-scope web ACL can exist nowhere else. Same Cloudflare step, same -c viewerCertificateArn= escape. 6. npx cdk deploy WpFrontend --profile wp-prod 7. Add the DNS records in Cloudflare, DNS-only (grey cloud), never proxied: wellness.greaterangelssoftware.com → the DistributionDomainName output, and origin.wellness.greaterangelssoftware.com → the AlbDnsName output. An orange-cloud record makes the path CloudFront → Cloudflare → origin, which breaks origin TLS validation and adds a hop nothing here accounts for. 8. Invoke the migration once, before anything serves: scripts/migrate-prod.sh (an aws ecs run-task against the MigrateTaskDefinitionArn output). 9. BUCKET=<SpaBucketName> scripts/publish-frontend.sh

The hostname is not a free choice. docs/saml/sp-metadata.xml pins the SP's entityID and ACS URL to wellness.greaterangelssoftware.com. To an IdP a new entityID is an entirely different service and the negotiated attribute release would not follow it, so once CSUB ITS has registered it, it can never change.

Rollback on AWS

One script per tier, and both have the same posture as scripts/rollback.sh above: never rebuild. A rollback that rebuilt could produce different bits than what previously ran and could fail for reasons unrelated to the release being reverted, which defeats the point of it as the fast path back to something known-good. Here it is a pointer move — every target is an immutable object the platform already holds.

CLUSTER=<ClusterName> SERVICE=<ServiceName> TASK_FAMILY=<ServiceTaskFamily> \
  scripts/ecs-rollback.sh                                       # one step back
... TARGET_REVISION=7 scripts/ecs-rollback.sh

It refuses a target carrying the same bits as what is running, and ends by fetching /api/version and matching the commit — because what a platform DECLARES is not what it SERVES.

The schema caveat. check_schema_is_current compares the live schema structurally against what the running code maps, and extra tables and columns are never an error. So rolling back across an ADDITIVE migration works, and rolling back across one that DROPPED or RENAMED something the old code still maps is refused — the release cannot start, and the deployment circuit breaker restores the newer revision by itself. Rolling the code back is not a way out of a destructive migration. Restore the database first (Database recovery below), then roll back.

The rollback is not durable. It moves the RUNNING thing; CloudFormation still declares the newer one, so the next npx cdk deploy — for any reason at all, including an unrelated change — puts it back. Each script prints the exact -c imageTag= command that makes it stick. Run it.

The SPA is a separate artifact and neither script touches it. If the release being reverted changed the frontend, check out that tag and republish: BUCKET=<SpaBucketName> scripts/publish-frontend.sh.

Drilling rollback on AWS

On every PR, hermetically. make test-api runs both scripts unmodified against PATH-shim aws and curl doubles (backend/tests/test_api_rollback_drill_bdd.py, US-86's four scenarios). The doubles model the PLATFORM, not the rollback — they know which revisions and versions exist and which will refuse to boot, and nothing about "the previous one" — so a script that picked the wrong target is faithfully obeyed and caught by the assertion rather than hidden by the double.

Live, by hand. Only one compute tier is deployed at a time, so only one can be drilled live; the other's deferral is recorded in docs/tier-deferrals/PASS-382-fargate-rollback-drill.md. Deploy two real releases, then run the tier's script for real and log it below. A tier with no row here has never been drilled live.

Date	Tier	Outcome	Duration	Notes
—	—	—	—	No live drill has been run yet.

Standing up staging

Staging is not a hand-maintained second copy of production's manifests — it is k8s-staging/kustomization.yaml, a kustomize overlay whose only resources: entry is the production base (../k8s). Every Deployment, Service, probe, and container port staging runs is therefore byte-identical to production's; the overlay is only permitted to patch three things:

Namespace — wellness-passport-staging, vs. production's wellness-passport.
Ingress hostnames — staging-demo.greaterangelssoftware.com and staging-docs.greaterangelssoftware.com, vs. production's demo.greaterangelssoftware.com / docs.greaterangelssoftware.com.
Image tag — staging carries its own images: block in k8s-staging/kustomization.yaml, re-pinned independently of production's, so the two environments can run different commits.

A second engineer stands up staging with the identical entry point that deploys production, scripts/deploy.sh, pointed at the overlay:

K8S_DIR=k8s-staging NAMESPACE=wellness-passport-staging \
  DEMO_HOST=https://staging-demo.greaterangelssoftware.com \
  DOCS_HOST=https://staging-docs.greaterangelssoftware.com \
  scripts/deploy.sh

This runs the same build → push → kubectl apply -k k8s-staging/ → wait for the migration Job → wait for rollouts → smoke-test → verify-publication pipeline production's own deploy uses (see the Bitbucket Pipelines' gated merge step), just against the staging namespace and hostnames. scripts/smoke.sh's DEMO_HOST/DOCS_HOST overrides and scripts/verify-publication.sh's matching overrides are what let the same scripts assert reachability against either environment's real URL.

The one one-time, human-run prerequisite (mirroring bootstrap/'s role for production) is that the wellness-passport-staging namespace needs its own gitea-registry image-pull secret before the first deploy, created out-of-band exactly as production's is — kubectl create secret docker-registry gitea-registry -n wellness-passport-staging ... — since that credential is deliberately never committed to the repository (see docs/secrets-management's local-secret-file convention).

Production vs. staging — the enumerated configuration differences

Read "production" in this section as the self-hosted cluster's production namespace, not as the AWS deployment. Since PASS-382 the two are different topologies: production is the AWS stacks above (S3/CloudFront, a selected compute tier, RDS), while k8s/ and k8s-staging/ are the demo and docs tier. The table below is about that cluster pair, and the "cannot drift" guarantee it states is a guarantee about them.

Per the structure above, staging differs from the cluster's production namespace in exactly these configuration values, and nothing else:

Axis	Production	Staging
Namespace	`wellness-passport`	`wellness-passport-staging`
Demo/API hostname	`demo.greaterangelssoftware.com`	`staging-demo.greaterangelssoftware.com`
Docs hostname	`docs.greaterangelssoftware.com`	`staging-docs.greaterangelssoftware.com`
Image tag	whatever the last production deploy pinned	whatever the last staging deploy pinned — independently

Every other manifest field — replica counts, probes, container ports, env var names, Secret keys, resource requests — comes from the one shared base (k8s/) both environments compose, so it cannot drift between them without also changing production. backend/tests/test_staging_deployment_bdd.py enforces this structurally on every make test-api run, by rendering both and diffing every Deployment outside these four axes.

Which commit is a pod running?

Every pod a deploy rebuilds carries the deployed tag as app.kubernetes.io/version, so the question is a label query rather than string-surgery on an image reference:

kubectl get pods -n wellness-passport -L app.kubernetes.io/version

NAME                        READY   STATUS    AGE   VERSION
api-79d9cbf8c4-xwq4f        1/1     Running   89m   0.1.0-3e64ac527ae1
docs-7bd54df4dc-8dbdx       1/1     Running   89m   0.1.0-3e64ac527ae1
frontend-5cbcbdf6fd-nwxqw   1/1     Running   89m   0.1.0-3e64ac527ae1
postgres-cd6979f56-d5brb    1/1     Running   26h

The tag is <VERSION>-<sha12>, so the trailing twelve characters are a commit: git log -1 3e64ac527ae1 resolves it in this repo. Because it is a real label it also selects, which is what makes it useful during an incident — find every pod still on a superseded build, across namespaces:

kubectl get pods -A -l app.kubernetes.io/version=0.1.0-fddb4b3d6f86

postgres is deliberately unlabelled and shows blank above. It is not built from this repo, and labelling it would put a changing value in its pod template — which, since it is emptyDir-backed, would make every deploy restart it and drop the database. k8s/version-stamp.yaml records that reasoning; scripts/deploy.sh rewrites the tag into it on each deploy, the same way it re-pins kustomization.yaml's newTag.

To ask a running process what it is — rather than what its spec claims, which differs if a tag was overwritten or a stale layer was served — the api serves its own stamp:

kubectl exec -n wellness-passport deploy/api -- \
  python -c "import urllib.request,sys; sys.stdout.write(urllib.request.urlopen('http://localhost:8000/api/version').read().decode())"

{"commit":"3e64ac527ae1d0f9...","version":"0.1.0","build_time":"2026-07-22T11:02:04-0700"}

It is public through the ingress too (https://demo.greaterangelssoftware.com/api/version), which is the quickest way to confirm what the edge is actually serving after a deploy. The SPA carries the identical three values — scripts/build-images.sh passes one commit/version/time triple to both images — so api and frontend disagreeing about their own provenance means something shipped out of band.

The namespace-level wellness-passport.io/last-deployed-tag annotation records what the last deploy pinned. That is the right input for rollback (below) and the wrong one for this question: it is a single value for the whole namespace, so it cannot describe a namespace mid-rollout or one still holding pods from an earlier release.

Rollback

scripts/rollback.sh restores the release that was running immediately before the current one:

scripts/rollback.sh                                   # production, one step back
K8S_DIR=k8s-staging NAMESPACE=wellness-passport-staging \
  scripts/rollback.sh                                  # staging, one step back
TARGET_TAG=abc1234 scripts/rollback.sh                 # a specific known-good tag

It reads the target namespace's wellness-passport.io/last-deployed-tag annotation — the same one scripts/deploy.sh stamps after every successful deploy — to find the currently-running tag, then walks the api Deployment's ReplicaSet history to the newest one that is not that tag, and redeploys exactly that tag through scripts/deploy.sh with SKIP_BUILD=1. Rollback never rebuilds: it always redeploys bits the registry already has and this pipeline already verified once, so a rollback cannot itself fail for a reason unrelated to the release being reverted.

Drilling rollback

Rehearsed against staging (never production) before this section is treated as load-bearing. This is now a runnable drill, scripts/staging-rollback-drill.sh (PASS-275), not a prose checklist — it performs exactly these steps and exits non-zero, naming what the staging URL is actually answering with, if the release does not revert:

Deploy a known tag to staging: K8S_DIR=k8s-staging NAMESPACE=wellness-passport-staging TAG=<tag-a> SKIP_BUILD=1 DEMO_HOST=https://staging-demo.greaterangelssoftware.com DOCS_HOST=https://staging-docs.greaterangelssoftware.com scripts/deploy.sh.
Deploy a second tag on top of it: ... TAG=<tag-b> SKIP_BUILD=1 scripts/deploy.sh. Confirm staging-demo.greaterangelssoftware.com answers with <tag-b> — the drill fetches the site and reads the provenance stamp it serves, and scripts/verify-publication.sh asserts the same during the deploy. This is an HTTP observation of what the URL answers, not a read of the api Deployment's declared pod-template image.
Run K8S_DIR=k8s-staging NAMESPACE=wellness-passport-staging scripts/rollback.sh.
Confirm the rollout completes and staging-demo.greaterangelssoftware.com answers with <tag-a> again — the previous release, restored, observed at the URL.

Run it against the real staging cluster: scripts/staging-rollback-drill.sh drives the real scripts/deploy.sh and scripts/rollback.sh against whatever cluster kubectl points at. Prerequisite: the drill deploys and reverts already-built tags — it never builds (SKIP_BUILD=1) — so you must first push two real staging releases and name them, along with the provenance commit each image carries (what the staging URL will answer with): TAG_A=<tag> STAMP_A=<commit> TAG_B=<tag> STAMP_B=<commit> scripts/staging-rollback-drill.sh. Without them the drill fails loud rather than deploying an unpushed placeholder that would ImagePullBackOff.

Database recovery

PR-B1 / TD-8 addendum (PASS-107) — the four scenarios in backend/tests/features/schema_recovery.feature, asserted by backend/tests/test_schema_recovery_bdd.py. This is schema recovery, not data recovery — see Backup and restore below for restoring lost rows from a dump. This section is for a database whose schema predates migrations entirely: no alembic_version table, missing a column the current code expects (a developer's pre-PASS-31 wellness_passport.db, or the PASS-59 Postgres demo volume provisioned before that story shipped).

Recognizing the failure

The app refuses to boot rather than starting silently broken, naming the missing column and the recovery command in the same error (app/schema_guard.py's ensure_schema). The deploy-time read-only check (python -m app.migrate check) refuses the same way — and, as of this story, no longer suggests make migrate for this case, since Alembic starts from base and would try to CREATE TABLE onto tables the database already has (OperationalError: table students already exists).

Recovery

make db-reset          # == python -m app.db_reset run

Destructive. This drops every table Base.metadata knows about (and any stray alembic_version), migrates to head from a clean slate, and reseeds the theme presets — and the demo challenge, when WP_SEED_DEMO=1 — exactly as a fresh checkout's first boot would. It refuses outright under WP_ENV=production: recreate-and-reseed destroys every row, which is never an acceptable recovery step against a real production database. A production database recovers from backup instead — see Backup and restore below.

Backup and restore

US-34 / PR-B3 — the three scenarios in backend/tests/features/backups_verified_restore.feature, asserted by backend/tests/test_backups_verified_restore_bdd.py. A backup that has never been restored is a hope, not a capability — this section exists so a launch does not ship on the strength of an unexercised script.

US-87 / PR-F4 (PASS-525) names this machinery, states an owner, and records a test cadence in one document for a CSU procurement reviewer: docs/continuity-and-recovery-plan.md. That document restates the RPO/RTO below rather than defining its own, and adds the one thing this section does not cover — the first full-recovery exercise against the deployed AWS production topology, still PENDING (docs/tier-deferrals/PASS-525-production-recovery-exercise.md).

Cadence

k8s/backup-cronjob.yaml runs daily (0 3 * * *, UTC) against the production database, writing into the backup-storage PersistentVolumeClaim (k8s/backup-pvc.yaml). backend/tests/backup_restore_conventions.py's max_gap_between_fires() parses that CronJob's own spec.schedule and computes the actual maximum gap between fires — a check against the deployed manifest, not a label trusted at face value. The PR gate enforces that computed gap never exceeds 24 hours.

Every run — success or failure — writes a *.manifest.json recording the outcome, start/finish timestamps, the dump filename, the table count captured, and (on failure) a populated reason. That manifest is what makes a failed backup visible to the operator: a monitoring job (or a human) reads it rather than trusting an unwatched exit code. scripts/backup-db.sh is the same procedure, runnable by hand from an operator's workstation (which has no in-cluster pg_dump but does have Docker) against any WP_DATABASE_URL.

Residual gaps, recorded rather than hidden:

backup-storage assumes a default StorageClass is provisioned on the cluster — an environment with none configured will fail to bind the PVC.
The PVC lives in the same cluster as the database it protects. A cluster- or node-level disaster takes both down together; an off-cluster backup destination (e.g. object storage) is not yet wired up.

Recovery objectives

Maximum acceptable data loss window (RPO): 24 hours — matches the daily backup cadence above; a failure between two runs can lose at most one day's check-ins. Expected time-to-restore (RTO): 60 minutes — the time budget for provisioning a scratch target, running scripts/restore-db.sh, and confirming the integrity checks pass, based on the restore drill below. Neither figure has yet been ratified by SHS; both are recorded as the numbers CI enforces (backend/tests/backup_restore_conventions.py's RPO / RTO) so an agreed figure, when it arrives, replaces a stated number rather than a blank — the same posture the Load-testing section above already takes on its own unratified targets.

RPO ratification status: PENDING (owed to SHS — see docs/tier-deferrals/PASS-464-shs-ratification.md). RTO ratification status: PENDING (owed to SHS — see docs/tier-deferrals/PASS-464-shs-ratification.md).

Restore drill

The documented restore procedure: run scripts/backup-db.sh against the source database, then scripts/restore-db.sh <dump-file> <target-url> against a scratch PostgreSQL target. restore-db.sh verifies pg_restore exited 0 and that the restored database's table count matches the dump's own table-of-contents count. The stronger claim — that the seeded rows survive the round trip, not merely that the tables exist — is asserted by querying the restored database's row values and counts directly; a pg_restore --schema-only run (which recovers every table name with zero rows) fails that assertion, which is the gap this drill exists to close.

Drill log

Date	Outcome	Duration	Notes
2026-07-25	Success	—	`make test-infra`'s restore-drill scenario (`test_backups_verified_restore_bdd.py::test_restore_drill_is_executed_before_launch`) run against two ephemeral `postgres:16-alpine` containers. `backup-db.sh` dumped a seeded source (1 student, 1 challenge, 1 task, 1 check-in); `restore-db.sh` restored it into a scratch target and passed its table-count check; the scenario's own row-level assertions confirmed all four seeded rows survived by value. Duration was not measured at the time — PASS-464 added the Duration column and the timed drill below.
2026-07-29	Success	5.03s	PASS-464: same scenario, now timed end to end (scratch container provisioning + `restore-db.sh` + the integrity queries), via `make test-infra`'s `rto:` line printed by `the_restored_database_passes_integrity_checks`. Well inside the stated 60-minute RTO — expected, since this is a 4-row ephemeral-container drill, not a production-scale restore; the figure is a floor, not a promise, and is recorded as a residual gap rather than hidden.

Security operations

US-44 / PR-F2 — the four scenarios in backend/tests/features/vendor_security_package.feature, asserted by backend/tests/test_vendor_security_package_bdd.py. The vendor security package itself — docs/hecvat-lite.md and docs/security-dast-baseline.md — lives alongside this runbook, not inside it; this section records the ongoing cadence, not the one-time assessment.

Dependency-audit gate

make security-deps runs on every change in the PR gate (bitbucket-pipelines.yml):

npm audit --omit=dev --audit-level=high — shipped frontend dependencies, failing the build at High or Critical.
pip-audit — every backend dependency, failing the build on any reported advisory (its own default posture, stricter than npm's floor).

A finding that cannot be fixed immediately (no upstream release, or the fix is a breaking major-version bump tracked as its own upgrade) is recorded in the accepted-risk register below with a rationale — never silently suppressed with an unrecorded --ignore-vuln or an unreviewed lockfile pin. As of this writing both audits are clean; the register has no entries.

DAST cadence

An OWASP ZAP baseline scan runs against the local docker-compose.yml stack (make security-dast) — see docs/security-dast-baseline.md for the procedure, the no-staging-tier caveat, and the full triage. Re-run:

before every production-bound release that touches routing, headers, middleware, or a new externally-reachable endpoint;
quarterly otherwise, so a finding introduced by a dependency bump (not a code change) doesn't go unnoticed between releases.

Authorization sweep

backend/tests/test_vendor_security_package_bdd.py's Scenario 4 runs on every make test-api — it is not a separate cadence, it is the PR gate. The route set it sweeps is derived from each route's dependency tree (backend/tests/vendor_security_conventions.py), not a maintained prefix list, so a new router under any path is swept automatically; an unguarded route fails the build the same as any other test failure.

QR secret rotation (PASS-97 / PR-C1)

US-35's "Exposed secrets are rotated and invalidated" scenario proves that rotating WP_QR_SECRET invalidates every token signed with the old value. This section is the "what do I do next" that scenario never covered. Note what this is not: the story that filed it assumed event QR posters were still static, unrotating tokens — that was true before US-9 shipped, but US-9's rotating token (app/services/qr.py, an iat/exp pair on every mint, re-minted on every admin-dashboard serialize) has been live since. There is no printed static poster to re-mint or re-print; a poster stops validating qr_ttl_seconds after printing regardless of any rotation. What a WP_QR_SECRET rotation actually costs is described below.

Credentials this secret signs

Four distinct credentials key off WP_QR_SECRET, all in app/services/qr.py:

mint_event_token — the staff LiveOps event QR a student scans to check in
mint_passport_token — the student's own My QR passport token
derive_short_code — the "WP-XXXX-NN" caption shown under My QR
derive_event_code — the typed 7-character event backup code (PASS-598)

A rotation invalidates all four at once — there is no independent per-credential rotation.

Restart required

get_settings() is process-wide @lru_cached (app/config.py). Editing WP_QR_SECRET in the managed secret store has no effect on a running process — every API worker keeps signing and verifying against the old secret until it restarts and re-reads the environment. An operator who edits the secret and sees nothing change has not failed; the edit simply has not taken effect yet.

What a post-rotation scan looks like

A token minted before the restart, still inside its rotation window, scanned after the restart comes back 400 with the generic invalid message — not the expired one — because the token's signature no longer checks out under the new secret at all; expiry is never reached. app/services/passport.py logs this at WARNING as event_qr_rejected reason=invalid_token, the same line a forged or foreign token produces. Expect a brief burst of these immediately after a mid-event rotation — it is the restart, not an attack.

Recovery window

A rotation costs at most 60 seconds of that burst — exactly qr_ttl_seconds (app/config.py, default 60 seconds): once every token minted before the restart has aged past its own exp, the invalid-token burst stops on its own with no further action.

No re-issue needed

qr_token and event_code are computed fields (schemas/challenge.py), never stored, so there is no re-issue affordance to build and none to run. The staff LiveOps screen already re-fetches its task list every 30 seconds (ROTATE_MS, LiveOps.tsx); the next scheduled re-fetch after the restart mints a fresh, valid token automatically, with no operator action beyond the restart itself.

Procedure

Generate a new secret value and update WP_QR_SECRET in the managed secret store (see "Secrets management & rotation" above for how JWT/QR secrets are provisioned).
Restart every API process so get_settings()'s cache picks up the new value — an env-only edit with the process left running changes nothing.
Confirm the restart landed, then expect the event_qr_rejected reason=invalid_token burst described above for up to 60 seconds; it is not an attack.
Take no other action: the staff dashboard and My QR both self-heal on their next periodic re-fetch, and there is no printed poster to re-issue or re-print.

Filed separately rather than folded in here: k8s/api-deployment.yaml never sets WP_QR_SECRET, so that demo topology signs with the denylisted dev default outside this production-only validator's reach; and there is no kid/previous-secret fallback, so a zero-downtime rotation (old and new secrets both accepted for one window) is not expressible today.

Accepted-risk register

Every accepted (not resolved) dependency-audit or DAST finding is recorded with its rationale at the point it was triaged — docs/hecvat-lite.md's VULN-1 row for a dependency-audit exception, or the per-finding table in docs/security-dast-baseline.md for a DAST finding — rather than duplicated here. As of this writing: no dependency-audit exceptions exist (VULN-1); all 15 DAST findings are Accepted with rationale (docs/security-dast-baseline.md), none above Medium risk.

Load testing

US-45 / PR-F3 — the three scenarios in backend/tests/features/event_scale_load_validation.feature, asserted by backend/tests/test_event_scale_load_validation_bdd.py. That module checks this section records a method, a scale, and results; that the targets published here are the same constants the scenarios assert against (backend/tests/load_validation_conventions.py); and that every result recorded here met its own target.

Date: 2026-07-20 · Build: feature/PASS-54-us-45-event-scale-load-validation

Method

An in-process load harness, run as part of the ordinary pytest suite rather than as a separate tier:

Cohort. 1,000 students and their enrollments are bulk-inserted straight into the database (seed_enrolled_students). The cohort is the fixture, not the subject — routing 2,000 HTTP requests to reach the starting line would add minutes and measure nothing under test.
Burst. 300 students sign in and each POSTs /api/checkins/scan with a freshly minted event token, driven sequentially through Starlette's TestClient. Only the scan itself is timed; sign-in is excluded, because a real event burst is students who are already signed in scanning a poster.
Report. A semester of check-in data is seeded funnel-shaped (everyone completes week 1, each later week retains ~85% of the one before) and GET /api/reports/participation is timed once, end to end, with small-cell suppression in force — privacy is on in production, so it is on in the measurement.
Percentile. Nearest-rank p95 (p95()), so the reported figure is a latency that was actually observed rather than an interpolation between two that were.

Reproduce with:

cd backend && python -m pytest tests/test_event_scale_load_validation_bdd.py \
    --log-cli-level=INFO -k "burst or responsive"

The load: lines in that output are the figures tabulated below.

Scale

Dimension	Figure	Source
Enrolled students	1,000	US-45 Description — fall scale
Check-ins in the burst	300	US-45 scenario 1 — "hundreds in a few minutes"
Burst arrival window	5 minutes	US-45 scenario 1
Check-in rows behind the report	4,523	1,000 students × 6 weeks, 85% weekly retention

Results

Three consecutive runs on the CI-class developer machine, 2026-07-20. The observed column is the worst figure across those runs, not the best.

Measurement	Target	Observed	Verdict
Check-in p95	400 ms	15.8 ms	Pass — 25× headroom
Participation report	2,000 ms	8.7 ms	Pass — 230× headroom

Supporting figures from the same runs: check-in p50 13.6–13.8 ms, max 72.6–76.2 ms (the max is the first request of the burst, which pays one-time import and connection setup). All 300 check-ins returned 200; none failed, and none was throttled. The 300-request burst completed in 7.2–7.5 s of wall clock, well inside the 5-minute arrival window it models.

Where the targets come from. 400 ms for a check-in is an interaction budget: the student is standing at a poster with the phone camera open, and the response should land before they lower it. 2,000 ms for the participation report is a dashboard-refresh budget for an aggregate over the whole cohort. Both are proposed here by the delivery engineer and have not yet been ratified by the SHS stakeholders — see the residual gap below.

Residual gaps

Recorded here rather than left implied, in the posture docs/a11y-manual-pass.md takes on its screen-reader gap.

This is not a production measurement. The harness runs in-process against a single-writer in-memory SQLite with no network, no Postgres, no concurrency, and no ingress. What it validates is that the application's own work does not grow badly with the size of the cohort — the failure this story exists to prevent. It does not tell you what the box will do on event night. A run against the compose stack (make up) or the k8s demo topology, with concurrent clients, is outstanding.
The targets are proposed, not agreed. Neither 400 ms nor 2,000 ms has been signed off by SHS. They are recorded as the numbers CI enforces so that the agreed figures, when they arrive, replace a stated number rather than a blank.
The rate limiter is per-process and in-memory (backend/app/rate_limit.py). Under N replicas the effective ceiling is N × the configured limit, and a student's burst is only throttled consistently if their requests land on one replica. At the current single-replica demo topology this is not reachable; it becomes real the moment the API is scaled out; production already runs two Fargate tasks (docs/diagrams/physical/production-aws.puml).
Concurrency is untested. The burst is sequential, so nothing here exercises lock contention, connection-pool exhaustion, or the uq_checkin_student_task race under simultaneous duplicate scans. The duplicate race is covered functionally by US-8's suite, but not under load.