Skip to content

Tier deferrals (manual passes owed)

Acceptance criteria that no automated tier in this repository can physically observe. Each is registered here with a named owner and a status; a PENDING entry does not block a merge, but it does mean a human owes a manual pass that nothing will fail if they skip.

Source: docs/tier-deferrals/*.md.

PASS-382 — the LIVE rollback drill against production

One deferred acceptance criterion from PASS-382 (Production topology: ECS Fargate, RDS PostgreSQL, S3/CloudFront). The tier's rollback is scripted and its script is reviewed; what has not been recorded is a run of it against the real account.

  • Criterion:

Scenario: Rollback restores the previous task-definition revision

— as a LIVE drill against wp-prod: deploy a known-good release, deploy a newer one, run scripts/ecs-rollback.sh, and confirm from /api/version that the older commit is serving again.

  • Unobservable because: it requires two real releases in ECR and a deliberate rollback of the production service, which no automated tier in this repository performs. The drill log table in docs/runbook.md (Drilling rollback on AWS) is where the live date lives; no row there means it has never been drilled live.

  • Owner: Anthony Graca.

What IS reviewed

  • scripts/ecs-rollback.sh never rebuilds or re-registers: it points the service at a task-definition revision ECS already holds, refuses a target carrying the same image digest as what is running, and ends by fetching /api/version and matching the commit.
  • The schema caveat is documented in docs/runbook.md: rolling back across a migration that dropped or renamed something the old code still maps is refused by the schema guard, and the deployment circuit breaker restores the newer revision. That is AWS's documented platform behaviour, not an observation of this account, which is exactly what the live drill would turn it into.

The Lambda tier this record once covered as a second row, together with its rollback script and the hermetic drill against fake aws doubles, was removed in PASS-733.

PENDING does not block the merge. It does mean the production rollback is owed one recorded live run by a named human, and that no automated tier will notice if that run never happens.

PASS-385 — a retracting URL bar leaves no band under the sheet

One deferred acceptance criterion from PASS-385 (Sign-in overscroll exposes the theme surface behind the fixed blue/white front door — give the app its own scroll container). Every other criterion in that story is gated by a test that reds under a named revert; this is the only one no available tier can observe.

  • Criterion:

Scenario: A retracting URL bar leaves no band under the sheet Given the sign-in content is shorter than the viewport And the mobile browser's URL bar is visible When the student scrolls and the URL bar retracts Then the sheet's bottom edge stays flush with the bottom of the viewport throughout And no band of any third colour appears between the sheet and the bottom of the viewport

  • Unobservable because: the criterion is about the divergence between the small viewport (100svh, URL bar showing) and the large viewport (100lvh, URL bar retracted), and about the live transition between them. Headless Chromium — the only browser tier this repository has — renders no browser chrome at all, so 100svh, 100lvh and 100dvh all resolve to the same number and there is no bar to retract. The scenario's own binding (test_app_shell_scroll_container_bdd.py, step "the mobile browser's URL bar is visible") probes those two units at runtime and fails on their equality, which is what the deferral rests on: not an opinion that the test is hard to write, but a measured fact about what this browser can produce. Playwright's set_viewport_size does not substitute — it resizes the layout viewport, so svh and lvh simply both change to the new value and stay equal.

  • Observing tier: a real mobile browser on a real handset, with visible chrome that retracts on scroll — Safari on iOS and Chrome on Android are the two that matter for this app's students. Per CLAUDE.md's mobile-first rule and PASS-385's own VERIFICATION section: 360px, 390px and 430px, in both a light and a dark semester theme, with the URL bar cycled.

  • Owner: Anthony Graca

  • Status: PENDING

  • Evidence: the owner will open / (sign-in) on a physical iOS Safari and a physical Android Chrome at each of the three widths, in the csub and harry-potter themes, scroll to make the URL bar retract and then reverse to bring it back, and record for each combination whether any band of a colour other than the sheet's #ffffff appears between the bottom of the sheet and the bottom of the viewport at any point during the transition — including mid-animation, which is where a dvh-versus-container mismatch shows most clearly. Screen recordings, not stills, since the criterion says "throughout". The recorded result replaces Status above with PASSED <ISO date> or FAILED <ISO date>.

What is already gated, and what this does not excuse

The app-shell change that this criterion is about is gated, twice over, by tests that do red under a revert:

  • app_shell_scroll_container.feature scenario 8 fails if any full-height screen is sized in a viewport unit while #root is sized by inset: 0 — which is the structural cause of the band. Reverting one route's min-height: 100% to 100dvh reds it.
  • Scenarios 1, 2, 3 and 7 drive a real compositor with elastic overscroll on and read the resulting pixels, so the general claim "no gesture uncovers the page canvas at either edge" is measured rather than argued.

What remains unobserved is narrower than the scenario text suggests: whether a real URL-bar retraction, specifically, can still open a band. The app shell removes the mechanism that produced it (nothing inside #root is sized in a viewport unit any more, and #root itself is inset: 0 on a position: fixed frame), but "the mechanism is gone" is a reading of the code, not an observation of a device, and this file exists so that distinction is not quietly lost.

PENDING does not block the merge. It does mean this story ships with a manual pass owed to a named human, and that no automated tier will notice if that pass is never done. Nothing in the test suite asserts on this file: writing it is what would make such an assertion pass, so it would gate nothing.

PASS-464 — obtaining SHS's ratification of the RPO/RTO figures

One deferred acceptance criterion from PASS-464 (Bind the stated RPO/RTO to measured behavior). Every other criterion in that story is gated by a test that reds under a named revert; this is the only one no available tier can observe.

  • Criterion:

Also record the ratification status of both figures with SHS once obtained.

The recording half is gated — see each_recovery_objective_records_its_ratification_status_with_shs in backend/tests/test_backups_verified_restore_bdd.py, which asserts docs/runbook.md carries a recognized RPO ratification status: / RTO ratification status: line for each figure. What is deferred here is narrower: the obtaining — SHS actually reviewing and agreeing to the 24h RPO / 60-minute RTO figures.

  • Unobservable because: SHS's assent to a specific number is a fact about an external human's decision, communicated outside this repository (an email, a meeting, a signed document) — no test in this suite, or any tier available to it, has anything to read until that communication happens and someone transcribes its outcome. A test cannot query SHS's agreement any more than it can query whether a meeting occurred.

  • Observing tier: a named human confirming directly with SHS (email thread or meeting) and recording the outcome.

  • Owner: Anthony Graca

  • Status: PENDING

  • Evidence: once SHS provides written sign-off (an email or meeting notes agreeing to the 24-hour RPO and 60-minute RTO figures, or naming different ones), the owner will: (1) update docs/runbook.md's two ... ratification status: lines from PENDING to RATIFIED (or update the RPO/RTO figures themselves, plus backend/tests/backup_restore_conventions.py's RPO/RTO constants, if SHS asks for different numbers), and (2) replace Status above with PASSED <ISO date>, citing the email thread or meeting notes as the record consulted.

What is already gated, and what this does not excuse

Both figures are gated on everything a test can observe without SHS's involvement:

  • schedule_fits_rpo() (bound by "the deployed backup cadence fits the stated data-loss window") checks the deployed k8s/backup-cronjob.yaml schedule's own computed gap against RPO directly — no longer against the independent MAX_BACKUP_INTERVAL constant that merely happened to agree.
  • restore_fits_rto() (bound by "the measured restore fits the stated time-to-restore budget", requires_tooling) times a real restore drill end to end and checks the measured wall-clock elapsed time against RTO.
  • missing_ratification_statuses() (bound by "each recovery objective records its ratification status with SHS") checks the runbook records a recognized status word for each figure — PENDING today.

PENDING does not block the merge. It does mean this story ships with a manual pass owed to a named human, and no automated tier will notice if SHS is never actually asked. Nothing in the test suite asserts on this file or on the runbook's specific status word beyond "is it a recognized one" — writing RATIFIED into the runbook without SHS's actual agreement would satisfy the test but not the criterion, which is exactly why the obtaining itself is deferred to a named human rather than encoded as a passing assertion.

PASS-525 — the first full-recovery exercise against production

One deferred acceptance criterion from PASS-525 (Name and document the BCP and DRP around the existing recovery machinery): "The first full-recovery exercise is performed against the production topology once it exists (see PASS-382) and its date recorded."

  • Criterion:

The first full-recovery exercise is performed against the production topology once it exists (see PASS-382) and its date recorded.

PASS-382/US-86 deployed the production topology on 2026-08-30 (docs/aws-production-deployment-log.md), so the "once it exists" precondition is met and the exercise is owed now. What is deferred here is narrower: performing the exercise itself and recording a real PASSED <date> or FAILED <date> outcome — see docs/continuity-and-recovery-plan.md's "First full-recovery exercise" section, currently PENDING.

  • Unobservable because: no CI tier in this repository holds wp-prod credentials — the same structural reason scripts/publish-frontend.sh is operator-run rather than CI-run. Beyond credentials, there is currently no supported observability path into the production database at all: docs/aws-production-deployment-log.md's §2.19 records that psql from a laptop is refused (PubliclyAccessible: false, isolated subnets, zero NAT gateways), ECS Exec is disabled (enableExecuteCommand: false, no ssmmessages endpoint), and the admin console is reachable only by a real CSUB SSO identity (there is no local login or break-glass account). The one working route — a hand-issued RunTask override on the migrate task definition — is an interactive operator action by construction, not something an unattended CI job can invoke.

  • Observing tier: a named human, running the documented restore procedure (docs/runbook.md's Backup and restore section) against the real production database (via the RunTask route above, or once enableExecuteCommand lands — see §2.19's "Proposed fix, not yet done"), and recording the outcome.

  • Owner: Anthony Graca

  • Status: PENDING

  • Evidence: once the exercise is performed, the owner will: (1) update docs/continuity-and-recovery-plan.md's "First full-recovery exercise" status line from PENDING to PASSED <ISO date> or FAILED <ISO date> and its date line to the ISO date the exercise was run, (2) record what was verified (a real production-shaped restore, checked by row value and count, not merely table names — the same bar test_backups_verified_restore_bdd.py's ephemeral-container drill already holds itself to), and (3) replace Status above with the same outcome, citing the exercise record consulted.

What is already gated, and what this does not excuse

Everything a test can observe without production credentials is gated by test_continuity_and_recovery_plan_bdd.py:

  • the plan document exists, names a real owner, and records a recognized exercise-status word (plan_owner() / exercise_status());
  • every dependency it names resolves to a real, deployed stack (dependency_stack_problems());
  • its RPO/RTO figures and ratification status agree with the CI-enforced constants and the runbook (declared_plan_objectives());
  • its stated objectives are pinned to the deployed single-AZ database configuration (infra_is_single_az() / plan_states_single_az());
  • every loss scenario cites a runbook procedure and script that are actually real (loss_scenario_problems());
  • the recurring restore test it points to still exists and is still wired into CI (restore_drill_scenario_exists() / restore_drill_wired_in_ci()).

PENDING does not block the merge. It does mean this story ships with a manual pass owed to a named human, and no automated tier will notice if the exercise is never actually performed. Nothing in the test suite asserts on this file or on the plan's specific status word beyond "is it a recognized one" — writing PASSED into the plan without actually performing the exercise would satisfy the test but not the criterion, which is exactly why the exercise itself is deferred to a named human rather than encoded as a passing assertion. test_continuity_and_recovery_plan_bdd.py's "The first full-recovery exercise is recorded" scenario is xfail(strict=False) for the same reason: it asserts the real claim (a dated PASSED/FAILED outcome) and is expected to fail until that claim is true, rather than being written to pass on a PENDING placeholder.