Business continuity and disaster recovery plan

US-87 / PR-F4 (PASS-525). This document answers HECVAT Lite's DOCU-01 (business continuity plan) and DOCU-02 (disaster recovery plan) questions — see docs/hecvat-lite.md's Business continuity & disaster recovery section, which points here.

Scope. One document for both BCP and DRP, per the story's own acceptance criteria: the pilot is one small, single-tenant application, and splitting "how the business keeps operating" from "how the system recovers" would duplicate the same owner, the same objectives, and the same dependency list twice. Most of what follows already existed before this document — US-32's infrastructure as code, US-34/PASS-464's automated, verified, objective-bound backup and restore, and PASS-382/US-86's deployed production topology. This document names it, states an owner, and records a test cadence; it invents no new mechanism and no new number.

Ownership

Owner: Anthony Graca

The owner is accountable for: keeping this document current as the deployed topology changes, obtaining SHS's ratification of the RPO/RTO figures below (tracked in docs/tier-deferrals/PASS-464-shs-ratification.md), and performing the first full-recovery exercise against production once scheduled (tracked in docs/tier-deferrals/PASS-525-production-recovery-exercise.md).

Dependencies

The pilot's continuity depends on three things outside this repository, each a real, deployed stack declared as CDK under infra/ (infra/bin/infra.ts) rather than a manually-provisioned resource:

Dependency	Real mechanism
AWS	member account 743050012859 (`wellness-passport-prod`), organization `o-7max4zgvjp` — see `docs/aws-production-deployment-log.md`
Image registry	Amazon ECR, repository `wellness-passport-api`, stack WpRegistry
Compute platform	ECS Fargate service, stack WpService
Database	Amazon RDS for PostgreSQL 16.14, stack WpData

Losing AWS itself (the account, or the us-west-2 region) is the top-level loss scenario every row in the table below is a narrower case of; the infrastructure-as-code dependency is what makes recovering from it a redeploy rather than a rebuild-from-memory — see the last row below.

Recovery objectives

Maximum acceptable data loss window (RPO): 24 hours. Expected time-to-restore (RTO): 60 minutes.

These are the same figures PASS-464 established and CI enforces (backend/tests/backup_restore_conventions.py's RPO / RTO) — this document restates them rather than inventing new ones, per this story's own scope boundary. They are stated against the deployed database configuration: single-AZ (multiAz: false, infra/bin/infra.ts, WpData) — a deliberate cost-driven deviation from ADR 0001 §5.3's Multi-AZ recommendation for production (infra/README.md). Under this configuration an Availability Zone failure is not an automatic failover; it is a restore, unavailable until an operator acts, which is exactly what the stated RTO budgets for. Stating this RTO against the ADR's Multi-AZ text instead would be the false-compliance hazard infra/README.md names — Multi-AZ's automatic ~1-2 minute failover is a materially different, faster recovery than what is actually deployed.

RPO ratification status: PENDING (owed to SHS — see docs/tier-deferrals/PASS-464-shs-ratification.md). RTO ratification status: PENDING (owed to SHS — see docs/tier-deferrals/PASS-464-shs-ratification.md).

Loss scenarios and recovery procedures

Each scenario cites the runbook section documenting the real, existing recovery mechanism — never an aspirational one — and, where applicable, the script that carries it out.

Loss scenario	Recovery procedure	Runbook section	Script
Data loss or corruption in the production database (bad write, accidental deletion, application bug)	Restore the most recent verified backup into the affected database, following the documented restore procedure	Backup and restore	scripts/restore-db.sh
Single-AZ RDS instance or AZ failure (no automatic failover under `multiAz: false`)	Restore from the automated RDS backup / point-in-time snapshot into a replacement instance; the stated RTO budgets for this operator-initiated recovery	Backup and restore	scripts/restore-db.sh
A database whose schema predates migrations entirely (a stranded pre-Alembic database)	`make db-reset` reseeds a clean schema from migrations — destructive, and refused under `WP_ENV=production`; a real production database recovers from backup instead, never from a reset	Database recovery	—
A bad deploy or crash-looping ECS service	Roll back to the previous known-good, immutable image already held in the registry — never rebuild	Rollback on AWS	scripts/ecs-rollback.sh
Total loss of the AWS account or region (the top-level case every row above narrows)	Redeploy the CDK stacks from source into a fresh account/region, then restore the database from the most recent off-instance backup	Production on AWS (PASS-382 / US-86)	—

Test cadence

Two tests, one recurring and automated, one a one-time full-rehearsal owed before this plan can claim to be exercised end to end:

Recurring restore test. The Restore drill is executed before launch scenario in backend/tests/features/backups_verified_restore.feature, asserted by backend/tests/test_backups_verified_restore_bdd.py, run via make test-infra — wired into bitbucket-pipelines.yml's nightly tooling- gated pipeline, not the PR gate (make test-api, which every PR runs, deselects it with -m "not requires_tooling"). This is the recurring restore test this plan's test cadence counts: pg_dump/pg_restore against a real Postgres instance, with the restored database's rows checked by value and count, not merely its table names. It has been run and recorded in docs/runbook.md's Drill log (most recently 2026-07-29, 5.03s, well inside the stated RTO — a floor from a 4-row ephemeral drill, not a production-scale promise).

Two caveats on "recurring," stated honestly rather than implied: Bitbucket has no in-file cron syntax, so nightly only fires automatically once a schedule is attached in Repository settings -> Pipelines -> Schedules — state this repo cannot see from the checked-in YAML — and until that schedule exists the pipeline runs only when a human triggers it by hand. And the step that runs it is fail-not-skip only when the tooling is present: a runner with no live docker daemon logs SKIP infra tests: docker daemon unavailable on this runner. and passes the step (bitbucket-pipelines.yml's *infra step), rather than failing over the drill not having run. 2. First full-recovery exercise against production. A rehearsal of the actual recovery path against the deployed AWS topology (PASS-382/US-86), not an ephemeral container — see below.

First full-recovery exercise

First full-recovery exercise status: PENDING First full-recovery exercise date: not yet performed

The production topology PASS-382/US-86 describes was deployed 2026-08-30, which satisfies the "once it exists" precondition this exercise was blocked on. It has not yet been performed: docs/aws-production-deployment-log.md's §2.19 records that there is no supported observability path into the production database at all today (no public access, no ECS Exec, no admin console reachable by anyone but a CSUB account) — the one working route is a hand-issued RunTask override, which is an operator action, not something a CI job can invoke unattended. Performing and recording this exercise is tracked as a tier deferral: docs/tier-deferrals/PASS-525-production-recovery-exercise.md. PENDING does not block this document from existing or from HECVAT naming it — it means the exercise itself, and updating this status to PASSED <date> or FAILED <date> with the recorded evidence, is owed to the named owner above.