Skip to content

Operational Readiness — Isolation, Recovery And Telemetry

  • Owner: NS5-012.
  • Procedure source and operational policy are recorded; target execution remains separate.
  • Product paths below are relative to the Product root, currently new-system/.
  • Existing owners: C1 tenant-composition.ts, the actual composition MySQL suite, Observability, and scripts/release/verify-deployment.mjs. No Control Tower store, scheduler or new writer.

1. Scope, Authority And Evidence

A covers tenant/privilege isolation, upgrade compatibility and available C1 receipt/applied-state reconciliation. B covers backup/restore/DR procedure, redacted logs, health and measured performance. A source PASS is not a target-role, backup, recovery or Production PASS.

The supplied recovery drill is six-table composition fixture recovery only. It copies the current definitions of T_CLIENT, T_MODULE, T_TENANT_MODULE, T_TENANT_CONFIG, T_TENANT_PROFILE_ASSIGNMENT and T_OPERATION_EVENT, inserts synthetic data, dumps that disposable schema and restores into another newly created disposable schema. It neither backs up production rows nor proves recovery of the whole Product, files, encryption keys, grants or providers.

Do not run a schema or role mutation on the basis of a read-only approval. Source/Git authorization is not Production/service/provider authority. Root Project Operations remains binding. Do not run CI, scheduled backups, cloud exporters, service restart, cutover, full-stack failover or master promotion as part of this procedure without their separately recorded exact authority.

2. Source Regression And Native Toolchain

From the Product root, using the installed repository toolchain:

pnpm exec tsx --test \
  packages/policy/src/isolation-upgrade.test.ts \
  packages/platform-administration/src/release-upgrade.test.ts \
  packages/db-mysql/src/platform-administration/tenant-composition-proof.test.ts \
  packages/observability/src/operational-readiness.test.ts
node --test tests/quality/operational-readiness.test.mjs
pnpm exec vitest run --config vitest.workspace.ts tests/contract/observability.test.ts
pnpm --filter @ctc/observability typecheck

Record revision, command, toolchain, failures and skipped cases. Historical 73-test and 32-test records belong to their original revision; never add them to today's executed test count. The quality test exercises the actual MySQL race interceptor against fake connections. It does not connect to MySQL. The Node HTTP test uses a disposable loopback server, not the Product service.

3. Actual MySQL Isolation, CAS And Lost-Acknowledgement Proof

Reuse tests/integration/tenant-composition.mysql.test.ts. Its existing target is deliberately specific: TCP 127.0.0.1, /prod as the schema-definition source, server identity kyle-mba:3306, and an absent disposable schema ns3_122_20260910. Do not weaken those checks to reach another DB. The source schema is read for definitions; mutations are in the disposable schema.

Load the approved connection into CTC_NS3_122_MYSQL_URL privately. Never paste the password into commands, logs, Task evidence or repository files. Confirm no other writer owns the disposable schema. An existing schema is a collision, not permission to delete/reuse it.

After the exact local fixture effect is authorized:

: "${CTC_NS3_122_MYSQL_URL:?Load the approved local connection privately}"
CTC_NS3_122_RUN=COMPOSITION pnpm exec vitest run --config vitest.workspace.ts \
  tests/integration/tenant-composition.mysql.test.ts

The two real race connections now pause after reading their CAS preimage, not at the removed FOR UPDATE. Preserve own-tenant positive paths, foreign receipt rejection, replay, stale/conflict behavior, audit rollback, lost commit acknowledgement recovery and exact readback. A race result must include two distinct writer connections and an independent observer. Retain the existing identical-operation replay expectation; do not change it merely to make a run pass.

Require NS3_122_SCHEMA_CLEANUP with the exact schema absent. An unset enable flag makes this suite skip; skipped tests are NOT_RUN, not actual isolation proof.

4. Isolated Backup/Restore And C1 Continuation

The additional restore case is opt-in. First confirm compatible mysql/mysqldump clients on the same approved local target. Optional CTC_NS5_012_MYSQL_BIN and CTC_NS5_012_MYSQLDUMP_BIN select explicit installed executable paths; do not install dependencies or start a service implicitly.

Set CTC_NS5_012_RESTORE_SLO_MS to the explicitly selected drill recovery-and-readback goal. There is no default and it does not establish a Production recovery SLO. Then run:

: "${CTC_NS3_122_MYSQL_URL:?Load the approved local connection privately}"
: "${CTC_NS5_012_RESTORE_SLO_MS:?Set the approved positive drill SLO in milliseconds}"
CTC_NS3_122_RUN=COMPOSITION CTC_NS5_012_RECOVERY_RUN=RESTORE \
  pnpm exec vitest run --config vitest.workspace.ts tests/integration/tenant-composition.mysql.test.ts

The case uses the existing MySQL adapter, not a simulated writer, for both profiles. It creates a private temporary dump (0600), records its SHA-256 and byte length, restores into a fresh ns5_012_restore_<random> schema, and compares ordered six-table row digests. It verifies current revision/operation/profile against receipt/history, rejects foreign operation lookups, continues one restored tenant from revision 1 to 2, retains its historical receipt, leaves the other tenant at revision 1 and proves the original fixture schema stayed unchanged.

backupMs measures dump creation. restoreAndReadbackMs measures fresh-schema creation, restore, row reconciliation and C1 readback; later continuation and cleanup are separately checked. Require both successful checks and verified cleanup; the final NS5_012_RECOVERY PASS is emitted only after cleanup succeeds. Do not interpret six synthetic tables as a Production throughput or RPO measurement. No customer workload is running during this fixed-data drill.

On failure, do not blindly retry writes. An uncertain schema-creation result remains an error for identity/readback review. Cleanup drops only a schema successfully created by this run, resolves fixture rows by their known UUID even after a lost insert acknowledgement, verifies child/client absence, closes pools and removes the temporary dump. It never drops a pre-existing schema. Do not print raw client stderr or dump contents. Cleanup failure keeps acceptance failed.

5. Target Roles Are A Separate Effect

The current capability-authorization.actual-route.test.ts is not a disposable-only role test: it modifies shared C1/C2 role assignments and SYSTEM grants, logs in a real configured C2 user and creates sessions. Do not automatically enable CTC_CAPABILITY_RUNTIME_TEST in the commands above. Use the NS3-107/NS3-122 owner evidence and explicitly authorize a named role fixture/copy before a new live run. Record tenant UUIDs, actual principals, scopes, allowed and denied commands, unchanged foreign readback and cleanup. Source contexts or a 200 response alone do not establish role proof.

Disposable local actual-role proof

After selecting the documented local fixture effect, use the disposable-only alternative:

CTC_NS5_012_ROLE_RUN=ISOLATED pnpm exec vitest run --config vitest.workspace.ts \
  tests/integration/operational-isolation.actual-route.test.ts

This uses the configured 127.0.0.1/prod connection after kyle-mba:3306 identity readback and the already-running 127.0.0.1:3000 service. It creates two random UUID tenants, role IDs 2/3 within each tenant and random-password users, then uses real login and HTTP authorization. It checks own-tenant positive read/role writes, foreign UUID/override rejection, DIRECT/SYSTEM access, revoked-manager self-escalation and foreign/unassigned/deleted role grants. Only its fixture rows are removed; existing C0/C1/C2 role/grant digests must remain unchanged. A private .generated/acceptance/NS5-012/role-isolation.json records safe checks/source hashes and cleanup. It does not mutate shared users/grants, bypass authentication or claim browser acceptance.

The native restore was executed on 2026-09-12. Keep --skip-add-locks in the dump options: --skip-lock-tables disables locks while dumping but does not remove restore-time LOCK TABLES statements. Restoring a fresh disposable schema needs no such privilege. Use --reporter=verbose --disableConsoleIntercept with the MySQL suite when retaining measurements; the agent reporter otherwise suppresses successful console evidence. Final 9/9 PASS and cleanup are recorded in NS5-012; this does not establish a Production backup policy.

6. Redacted Diagnostics And Manual Performance Baseline

Observability retains its existing field allowlist, best-effort exporter behavior and bounded metric registry. Unknown error names become Error; built-in timeout/cancellation names and safe error codes keep their classifications. Nested/nonfinite values do not pass the primitive field contract. Caller-supplied free text in an allowed string field is still the caller's responsibility; this is not an arbitrary text/PII scanner. Durable support audit remains the separate support owner.

With an explicitly approved existing HTTP target and independently known release/environment:

: "${APPROVED_RELEASE:?Use the intended release, not a value blindly copied from the response}"
: "${APPROVED_ENVIRONMENT:?Use the approved target environment}"
node scripts/release/measure-readiness.mjs \
  --origin=http://127.0.0.1:3000 \
  --expected-release="$APPROVED_RELEASE" --expected-environment="$APPROVED_ENVIRONMENT" \
  --samples=20 --interval-ms=250 --timeout-ms=5000 \
  --output=.generated/acceptance/NS5-012/readiness-baseline.json

This does not start the port-3000 service. Use the existing service contract if service work is separately authorized. The sampler reuses verifyDeployment for all identity, no-store, liveness and readiness decisions. Each sample issues exactly three read-only requests; redirects are not followed. Sampling is bounded (1–100), manual and unscheduled.

The private report stores only expected identity, counts, whole-response durations and min/mean/ p50/p95/max (nearest rank), HTTP/transport failure counts and a report hash. It does not persist raw response bodies, cookies, observed IDs, error names or provider payloads. Any failed sample fails the report. A low latency does not override identity/readiness failure. This measures the three diagnostic endpoints, not business-page latency, query budgets or production capacity. Use comparable revision/environment/load/sample settings for comparison; do not infer an SLO from one small run. No external exporter/backend has been selected or activated by this tool.

7. Full Operational Recovery Policy

For a real incident/DR drill, retain the parent deployment/recovery runbook's sequence: identify and contain the incident, preserve the recovery point, keep provider/scheduler/physical effects quiescent, provision an approved isolated target, restore the admitted DB/file/key/artifact set, reconcile UUIDs/relationships/receipts and actual roles, verify release/health/readiness, and obtain separate traffic/writer/provider failover authority. Failback needs its own state reconciliation; do not replay unknown external effects or copy an old source PASS as recovery evidence.

The owner confirmed this policy on 2026-09-13:

  • create one complete database recovery anchor daily and hourly incremental/binlog recovery points;
  • retain those recovery points for 35 days;
  • target Production RPO 1 hour and RTO 4 hours;
  • recover the full stack: database, matching application release/artifacts, configuration, keys/secrets references and required provider state;
  • run an isolated recovery drill quarterly;
  • use the existing redacted structured logs as the current telemetry baseline. External exporter selection and activation are deferred.

The daily full backup covers the complete database dataset at that recovery point. Application source/artifacts, configuration and provider material are preserved and restored through their own approved stores as part of the full-stack recovery set; they are not embedded in the database dump. The local six-table drill does not prove this Production policy has been scheduled or exercised.

Use PASS / FAIL / BLOCKED / NOT_RUN / NOT_APPLICABLE per requirement. NS5-012 local acceptance is complete. Production scheduling, retained-backup readback, full-stack restore/failover and any external exporter remain NOT_RUN until separately authorized and executed. Never average passing source tests into an overall Production PASS.

---

Source: new-system/docs/runbooks/OPERATIONAL_READINESS.md — ctc repository · This page is an English translation of the source document. Identifiers, file paths and code symbols are unchanged.

In this section

Document Contents
OPERATIONAL_READINESS Isolation, Recovery And Telemetry
README Runbooks