ADR 0028 — Persistent Release And Runtime Receipt Contract¶

Date: 2026-09-12
Status: DECIDED DESIGN / IMPLEMENTATION NOT STARTED
Request: NS5-007 A, the Cloud 2 design assignment, followed by the direct fix it then.
Owns: CTC release-consumption contract, selected storage integration, revision/idempotency, unknown-result reconciliation and bounded compensation. Not a live deployment/migration receipt.
Task: NS5-007. Preserves 0027.
Source baselines: CTC 379fb8629c26303c73bbc2881abe006beaf5c5ef; Control Tower a7cd1a66df50fea4f670a62227ff8e20e57824d8.

1. Decisions replacing the review's unresolved choices¶

Decision	Selected contract	Explicit non-goal
A-D1 source ownership	Control Tower owns desired releases; CTC NS3-122 owns applied settings and receipts.	No competing release origin in runtime config or a new service/database.
A-D2 physical storage	Two typed tables, `config_release` and `config_release_journal`, in the existing Control Tower operational SQLite database, through its existing store owner.	Not fields inside the mutable `control_plane_state.payload_json`; not migration execution.
A-D3 execution identity	Operator-triggered execution under the existing authenticated CTC Runtime session; Control Tower identity is separate attribution. Trusted target readback is required to confirm an effect.	No autonomous publisher, impersonation, new machine account, credential persistence or resurrection of retired Control Tower write proxies.
A-D4 mixed areas	One `composition` area for one target's profile/module/config change; entry kinds remain domain-specific. One existing writer invocation, never one transaction per area.	No capability/credential/template writer inferred from lifecycle enum values.
A-D5 inverse scope	Version 1 guarantees compensation only for previously explicit, still-valid operational values. Prior absence, offering termination and unsupported inverses produce `NO_SUPPORTED_INVERSE` before dispatch.	No invented delete/reset, fallback-to-default or partial operation labelled full rollback. Existing basic Admin/B1 flows are unchanged.
A-D6 concurrency	Replace the existing composition owner's unconditional revision upsert/write lock with tenant-scoped compare-and-swap. Keep its transaction and receipt owner.	No new MySQL revision table, `FOR UPDATE`, second writer or source fix claimed by this document.

These are selected engineering decisions, not a request to choose the same options again. Database migration execution, actual target/principal configuration and runtime acceptance are subsequent effects with their own authority. The original assignment is design, not permission to alter either live database. B1 remains delivered at its original SOURCE evidence ceiling.

2. Existing owners and why storage is separated¶

Current CTC sources are lifecycle, composition parser, applied-state writer, Admin transport and B1 codec. The tenant-config boundary and Control Tower boundary assign desired configuration to Control Tower while ordinary runtime requests continue from local applied state.

The inspected Control Tower files are:

Path in `kylejang1/control-tower` at the baseline above	Observed responsibility
`scripts/lib/control-plane-sqlite-store.mjs`	Existing SQLite connection, migration ledger, singleton state and atomic `updateState`.
`scripts/lib/control-tower-sqlite-state-store.mjs`	Operational-state wrapper; both full-state `save` and transactional `update` exist.
`scripts/ctc-control-tower-static-server.mjs`, lines 54–97	Resolves the existing `CONTROL_TOWER_CONTROL_PLANE_DB_FILE`; uses full-state persistence and an in-process mutation lock.
`docs/plans/CT-003-config-release-lifecycle.md`	Retired direct-write/metadata-only release routes return 410. No active release executor is established by that document.
`tests/control-plane-sqlite-update.test.mjs`	Existing store update tests, not release publication acceptance.

A stale saveState(oldSnapshot) can erase a field committed by a later updateState. This was reproduced on the exact existing store with two local connections and disposable data; it is not a claim that a live release was lost. Moving every unrelated state writer to a new protocol would expand this task. Dedicated tables in the same database isolate release records from those whole-state saves without creating another database or semantic store owner.

Do not seed releases from old runtimeConfigDrafts, mapping labels, snapshots or Task records. Those are not authenticated applied-release evidence. No source-only test upgrades them. The dated August CT-003 CTC dependency assessment predates September NS3-122; retain its no-bypass principle, not its old CTC status as a permanent veto. CT implementation must update its own owning documentation when it consumes this contract; this CTC design does not silently activate CT-003 or change its retired routes.

3. Selected physical storage contract¶

3.1 Location and owner¶

Use the existing database selected by CONTROL_TOWER_CONTROL_PLANE_DB_FILE and the existing ControlPlaneSqliteStore migration/connection owner. Do not hard-code a second DB path, add a repository/package/service, or create a separate ReleaseStore framework. Add bounded release methods to that owner when implementation is admitted. Keep release rows out of loadState, saveState, updateState, state seeding and generic state import/export.

Use SQLite STRICT tables. The following is the selected field/index specification, not an executable migration. The migration belongs to Control Tower's existing migration ledger; its number is the next unused version at implementation, never an assumed production version.

3.2 `config_release`¶

Columns	Type and constraint
`release_uuid`	TEXT primary key; canonical UUID checked by the decoder; never a CTC numeric ID.
`deployment_ref`, `environment`, `tenant_uuid`	Non-null TEXT; exact registered deployment/environment and canonical tenant. Immutable after creation.
`schema_version`, `binding_version`	INTEGER, exactly 1 for this decoder. Unknown versions are unavailable, not coerced.
`release_version`, `base_version`	INTEGER within JavaScript safe-integer range; base >= 0, release > base. Allocation is serialized in the same store transaction.
`base_release_uuid`	Nullable same-target release reference; null only when there is no prior release lineage.
`row_revision`	INTEGER, starts at 1, increases exactly once with every committed row/journal mutation; <= 9007199254740991.
`lifecycle_state`	TEXT constrained to the existing eight lifecycle states. No new Task status.
`definition_json`, `definition_hash`	Strictly decoded release definition and SHA-256 digest; editable only in DRAFT, immutable from validation onward.
`author_ref`	Non-null authenticated Control Tower actor reference, not a Runtime user UUID.
`approval_json`	Nullable typed approval evidence: actor, timestamp, definition hash and explicit high-risk confirmation where required. No new two-person rule.
`operation_uuid`, `runtime_actor_uuid`, `request_hash`	All null until publish preparation, then immutable as a group; UUID/UUID/SHA-256 TEXT respectively. `operation_uuid` is globally unique in this store.
`request_json`	Nullable exact normalized Runtime command from preparation; fixed expected revision, reason, actor binding and values. No credentials or session bytes.
`execution_outcome`	TEXT: `NOT_REQUESTED`, `UNCONFIRMED`, `APPLIED`, `NOT_APPLIED` or `INTEGRITY_ERROR`. This is evidence, not a lifecycle state.
`receipt_json`, `observed_json`	Separate nullable versioned safe projections. Past applied result is never overwritten with the current snapshot.
`compensates_release_uuid`, `compensated_by_release_uuid`, `superseded_by_release_uuid`	Nullable same-target links, no self-link/cycle; confirmed compensation/supersession pointer changed only by CAS.
`created_at`, `updated_at`	UTC TEXT timestamps for attribution/order; not concurrency tokens.

definition_json contains the target binding, exact parsed operational changes, approved reason, base Runtime revision string, presence-aware preimage for touched keys, profile token, definition versions and separate source-ref/runtime-artifact evidence. Preserve JSON types and explicit empty values. Project only admitted fields; never spread a Runtime snapshot/config row into this JSON. Use the existing parser limits: six policy keys, 100 module keys and the current versioned profile. Reject unknown fields, duplicate keys, invalid values, secrets and raw error payloads.

Required indexes/relations: unique (deployment_ref, environment, tenant_uuid, release_version); index (deployment_ref, environment, tenant_uuid, lifecycle_state, updated_at); unique operation_uuid; unique (release_uuid, deployment_ref, environment, tenant_uuid) for scoped foreign-key references. Same-target self references use the latter composite key with restricted delete, not cascade. Allocate a new release version above all allocated versions for that target; base_version still identifies the chosen prior lineage, not the preceding abandoned draft. Overflow rejects before any write. Multiple compensation candidates may refer to an original; only one successfully verified full compensation can acquire its compensated_by_release_uuid.

3.3 `config_release_journal`¶

Columns	Type and constraint
`event_uuid`	TEXT canonical UUID primary key.
`release_uuid`, `deployment_ref`, `environment`, `tenant_uuid`	Non-null scoped foreign key to `config_release`, restricted delete.
`sequence`	INTEGER > 0; equals the resulting release `row_revision`; unique `(release_uuid, sequence)`.
`command_uuid`, `command_hash`	TEXT UUID and SHA-256; command UUID globally unique; checked before CAS for replay.
`event_kind`	Typed creation/edit/validation/impact/approval/publish-intent/readback/observation/compensation/supersession event.
`actor_ref`, `recorded_at`	Authenticated actor and UTC time.
`from_state`, `to_state`	Valid lifecycle values; nullable from-state only on creation.
`evidence_json`	Versioned, event-specific redacted payload and original command result; no arbitrary JSON bag.

Create/edit/transition/observation writes use one short BEGIN IMMEDIATE transaction: read a previous command by UUID; verify matching scope/hash or conflict; check expected row revision and allowed transition; conditionally mutate the row and increment revision; insert one journal record; commit. Row mutation and journal insertion succeed together. Failed CAS changes neither. Identical command replay returns its recorded result plus a separately labelled current record. Network/Runtime calls never occur inside this transaction. Busy/timeout is not successful commit; reread the same command UUID before retrying. Do not use the static server's process-local promise lock as a cross-process guarantee.

3.4 Retention, reset and backup¶

No automatic deletion, soft deletion, TTL or recent-N truncation of release/journal idempotency records in version 1. Old general Control Tower receipt limits are not reused. A future bounded retention change must keep replay identity and unresolved/compensation links; no silent expiry. The same applies to Runtime TENANT_OFFERING_COMPOSITION_APPLIED receipts relied on by releases: exclude them from generic notification cleanup before enabling publication and prove it in tests. The existing history API's latest-100 projection is neither storage retention nor proof of absence.

Back up release rows and journal as a consistent part of the existing SQLite database, and preserve Runtime receipts with the corresponding Runtime backup. Test recovery from each side independently. Restoring an older Control Tower backup cannot certify Runtime rollback; reconcile old operation IDs first. A restored/reset Runtime database is not a normal retry target even when tenant UUID and revision strings match. Invalidate outstanding target bindings and reconcile recovery explicitly; never resend from a pre-restore approval against a replacement database by matching its hostname.

4. Runtime CAS correction inside the existing owner¶

Keep applyTenantOfferings as the sole configuration/revision/receipt transaction owner. The selected correction does not require a new MySQL table or a second revision counter. DB_QUERY_RULES forbids FOR UPDATE; do not copy it into release persistence or preserve a test assertion as policy.

Authenticate and resolve the displayed tenant/actor exactly as today. Parse and normalize once using the current parser/hash owner. Existing same-operation receipt lookup precedes mutation.
Start an explicitly consistent transaction for the operation; do not rely on pool defaults. Select the existing tenant identity with FOR SHARE to preserve its lifetime, not to serialize all configuration writers. Use one REPEATABLE READ view for the raw composition row and all before-state operational values. The existing rule forbids FOR UPDATE, not FOR SHARE.
Read the physical composition row (including deletion marker) by the existing unique tenant/key predicate. Retain its exact encoded preimage separately from the parsed effective snapshot. Validate the expected revision/profile token and all current parser/definition rules.
For an active row, conditionally update that exact tenant/key only if its stored preimage still matches byte-for-byte and its deletion marker is unchanged. Store the next revision and resulting assignments in that same update. Use the tenant/key index first; do not wrap indexed columns in functions. Any required byte comparison is on the non-indexed payload, subject to actual schema verification. A future executable statement must be checked against the real column types.
A physically missing row uses insert-if-absent, not an overwrite-on-duplicate upsert. A duplicate means the competing operation won: roll back and reconcile its own operation receipt, or return conflict. The old first-assignment path remains supported. A physically soft-deleted row uses a conditional revival of its observed payload/deletion marker; never treat it as an unobserved gap. Advanced release preparation must not treat a reset/revived composition counter as continuous historical evidence. This does not turn existing basic Admin compatibility into a new denial.
Zero matched rows is a stale preimage conflict. Exactly one successful revision claim is required before writing the operational values and inserting the receipt. Because revision always changes, do not use a no-op update as the claim or assume a driver's matched/changed-row mode arbitrarily. Reject the next revision above 9999999999999999999 before writes; use decimal strings/BigInt.
Apply the existing module/config/profile changes, read after-state and insert the same operation receipt in the same transaction. Failure rolls all of them back. Preserve definition validation and tenant scope; verify catalog changes and same-profile cases in the actual concurrent test.
Preserve existing commit-unknown behavior: destroy uncertain connections, never retry commit on another connection, reread the same receipt and then take a fresh authoritative readback.

A competing writer's changed revision makes the conditional write fail even if earlier snapshot reads were successful. Do not assume a nonlocking SELECT serializes writers. Keep the parent-tenant shared-lock ordering consistent with retained compatibility callers; do not remove guards across other owners as incidental cleanup. All composition entry points must continue to delegate here. Before rollout, test old/new writer overlap or drain old in-flight writers; untested mixed artifacts are not declared safe. No CAS source change or real MySQL interleaving was executed by this ADR.

5. Revision, normalization and lifecycle binding¶

Release identity/version, release-row CAS revision, document schema version, Runtime expected revision, receipt applied revision, current Runtime revision and profile token are independent. No Runtime revision is converted to Number. A release may have version 17, apply from 42 to 43, and later be observed at current 47. Its receipt stays 43; compensation can create 48, not 42.

The approval digest binds target, typed values, presence-aware preimage, approved reason, source and runtime evidence, and baseline. Specify schema version 1 and a deterministic canonical JSON encoder: recursively sort object keys, preserve array order after domain normalization, preserve JSON types, reject undefined/nonfinite values, UTF-8 encode without whitespace, SHA-256. Test cross-repository golden vectors. This digest is not the existing writer request hash.

Runtime request normalization/hash remains in the existing composition owner. Reuse its exact sorting/omission rules; do not reproduce them in Control Tower. Preparation returns the canonical request hash with the authenticated Runtime actor UUID. Neither retry time nor retry count is appended to reason. Changed actor, target, expected revision, reason or values requires a new logical request, not reuse of the old operation UUID.

Bind typed operational data to the existing pure lifecycle, with the selected additive composition area. Derive high-risk kinds on the server, not from a caller's kind label. Publish and rollback helpers remain pure markers: invoke/persist them only after the external proof condition below. Validation/impact require real parser/preimage evidence; entry count is not business-impact assessment. Both healthy and unhealthy observations can be OBSERVED.

CTC remains the single source owner of lifecycle semantics. If Control Tower consumes that core, ship a versioned compiled artifact generated from the existing module with its source SHA/digest, not a hand-maintained copy, second implementation, deep source import or new registry/framework. Artifact installation is a later explicit build/release effect. Pin the artifact with the binding version and reject incompatibility; this ADR neither publishes a package nor changes exports.

6. Execution and proof contracts¶

The initial execution path is an operator-initiated, authenticated CTC Admin request. Use the existing SYSTEM check and existing per-section capabilities. The authenticated Runtime session supplies runtime_actor_uuid; a Control Tower username, release author or body field never does. Approval records intent, not a new role grant. Recheck current execution rights; do not impersonate an unavailable or revoked actor for a retry. Another authorized actor may reconcile read-only.

Control Tower's retired direct-write and metadata-only release endpoints remain 410. Do not replace them with an autonomous service-account proxy. This design selects explicit operation preparation/execution/reconciliation ports; implementing those ports is not evidence that a production endpoint/principal is already configured. Ordinary business requests never call CT.

Extend the existing Admin preview/readback owners with narrow release-proof projections when implementation begins; keep ordinary NS3-122 and B1 request/response shapes compatible:

Operation	Required server-owned result / rule
Prepare	Validated target binding, authenticated actor UUID, exact normalized command, request hash, base revision/profile token, touched-key presence/preimage, actual source/artifact identity and safe preview. No apply.
Execute	Identical operation UUID and prepared command through the same `applyTenantOfferings`; no new SQL writer or implicit target selection.
Receipt read	Trusted target/environment, tenant UUID, operation UUID, event type, actor UUID, request hash, expected/applied revisions and allowlisted before/after delta, plus read time.
Current observation	Current revision and current touched-key projection with runtime identity/read time; separate from the historical receipt.

The existing operation GET only returns operation UUID/applied revision alongside the current snapshot. It is insufficient to authenticate every prepared-request field, so the same DB owner must provide the narrow additional proof, without exposing raw audit JSON, credentials or other configuration keys. Same-operation reuse with different hash/actor is a conflict.

Only a configured target-authoritative read provider may confirm receipt evidence. It must use an authenticated Runtime read under existing read authority and a trusted deployment mapping; arbitrary URLs, browser-submitted receipt JSON, log text, hashes without trusted origin and Control DB labels are not proof. A client-carried result can be stored as unverified observation only. The implementation must bind the provider to the authorized target before enabling publication; this is environment configuration, not authority to create credentials or a new identity model. No live target or machine principal is selected by this design document.

7. Publish, duplicate requests and unknown results¶

Persist the fixed publish intent/operation under row CAS in CT transaction C1, commit, execute Runtime R outside that transaction, obtain target-authoritative proof, then commit receipt linkage and lifecycle publication in CT transaction C2. There is no distributed transaction across C1/R/C2. Persisting intent sets UNCONFIRMED; it does not set PUBLISHED. Two dispatch attempts must carry identical actor and command, so the existing Runtime operation receipt arbitrates the one effect.

Observation	Recorded outcome and next action
Preparation/intent cannot commit	No Runtime dispatch; reread the same CT command ID if local commit is uncertain.
Intent committed, crash before/around send	Remain UNCONFIRMED; inspect the same operation ID. Do not infer never sent.
Same command ID/hash	Replay original CT result. Same Runtime operation/hash/actor reuses its receipt, not another revision.
Same ID with changed scope/body/hash/actor	Conflict; no additional effect or silent rebind.
Runtime commit response lost, matching receipt found	Mark APPLIED with that receipt; current state may already differ.
Runtime applied, CT C2 failed	Reconcile and retry C2 only; no new operation or compensating write.
No receipt, timeout, denied read, unavailable/corrupt state	Remain UNCONFIRMED or INTEGRITY_ERROR; never label failed from absence alone.
Trusted definitive pre-mutation rejection or confirmed rollback	NOT_APPLIED only when the execution channel also excludes another outstanding send of that intent. Otherwise UNCONFIRMED.
Receipt actor/hash/tenant/revision mismatch	INTEGRITY_ERROR, preserve evidence, no publish/rollback marker.
Current revision newer than receipt	Keep historical APPLIED and separately evaluate touched-key drift; do not rewrite the receipt.

Do not queue automatic retries, regenerate operation IDs, increase expected revision, change actor or re-preview an old command while its outcome is unknown. A deliberate retry may use the exact original request under the same still-authorized Runtime actor; reread first. A late delivery and a missing receipt are compatible facts, so absence is not cancellation. Supersession of an unrequested candidate is possible; after intent exists, require proven quiescence/resolution before superseding. A lifecycle supersede label cannot fence a Runtime request that has already been delivered.

8. Compensation and rollback completion¶

Version 1 is an explicitly compensatable operational release, not a general tenant clone/reset. An absent setting, an unassigned module and a missing profile are not equivalent to an explicit empty string, false, or default profile. Store presence independently. NO_SUPPORTED_INVERSE is an advanced release preparation result for such touched baselines; it does not block normal Admin configuration, B1 onboarding or unrelated reversible releases.

For a supported compensation: read current Runtime state and valid definitions; compare each original touched key with the original committed after-value/presence. Changed touched fields produce conflict instead of overwriting another operator. Untouched fields and all offering assignments remain unchanged. Create a new compensation release with a new operation ID, CURRENT revision/profile token, original before-values and an explicit reason; preview/approve/apply via the same owner. A stale compensation baseline must be re-reviewed, not silently rebased.

Confirm the compensation's receipt AND its full inverse delta before marking the original ROLLED_BACK. Preserve both receipts, increasing Runtime revisions and original audit. Subsequent drift is a separate observation. If no writes are needed because someone else independently restored values, record that observation but do not invent a compensation receipt or mark a release-owned rollback. If only part of the inverse is supported, do not apply a partial restore and report full success. A forward-fix release may be separately reviewed and linked as superseding, not rollback.

Thus prior absence/reset and offering termination are resolved as out of version-1 automatic compensation, rather than left as permission to invent inverse operations. A later domain-owner extension can add them with its own requirements; A storage and reversible publication need not wait.

9. Implementation order and acceptance¶

Implement and test the existing composition owner's CAS and proof projection first, keeping its public positive/retry behavior and all Admin/B1 callers. No MySQL migration selected here.
Admit the two-table migration and bounded methods in the existing CT store; update CT's own CT-003/schema documentation in that source change. Do not auto-run a migration on a live host or through normal server startup merely because this ADR exists.
Bind the single lifecycle artifact, exact target and authenticated operator/read provider; implement explicit preparation/publication/reconciliation/compensation. Keep retired proxies off.
Run native source tests and authorized disposable-target crash/concurrency/restore drills. Only their actual results can advance persistence or runtime acceptance.

Required acceptance, not executed by this document:

Cases	Required result
Store CAS competition; row/journal failure injection; stale full-state save	One mutation, inseparable journal, release tables unaffected by generic state writes.
CT command and Runtime operation duplicates, including changed actor/reason	One effect for exact replay; mismatched reuse conflicts.
First-row insert race, active/soft-deleted preimage CAS, concurrent Admin/B1 edits	No lost update, no blind revival, no cross-tenant changes; current positive flow retained.
Mixed profile/module/config failure and revision overflow	All-or-nothing settings/revision/receipt; overflow rejected before write.
Lost commit, missing/denied/corrupt proof, C2 failure, delayed request and supersede	Unknown is retained, same-ID reconciliation, no automatic resubmit/false cancellation.
Receipt at 43 with current snapshot at 47	Historical and current evidence remain separate.
Explicit empty/false versus absent; unsupported offering inverse	No default materialization or unapproved delete; preparation states exact inverse limit.
Compensation with concurrent touched/untouched edits and lost response	No overwritten touched drift, unrelated fields preserved, distinct verified compensating receipt.
Secret-field injection, arbitrary target, forged actor/proof	Refuse unsafe input without secret echo or identity/target delegation.
Receipt retention and independent CT/Runtime backup restore	No expired-ID duplicate effect, old target binding cannot replay into restored/replaced state.
Actual permission revocation, lifecycle-artifact mismatch, Control Tower outage	No impersonation; no false publication; ordinary local business/config flows keep their boundaries.

10. Evidence, consequences and rollback¶

This decision run executed the exact two existing Control Tower store tests and two diagnostic interleaving probes against blob-verified source with Node 22.16.0 and disposable SQLite files: PASS 4/4. One probe intentionally reproduces stale full-state overwrite; PASS means the unsafe alternative was demonstrated, not fixed. Disposable SQLite directories were outside both repositories and were cleaned up; test source and logs remain in the session evidence artifact. There is no new release-store implementation, MySQL test or target-role acceptance.

The selected tradeoff is two typed tables in an existing owner instead of refactoring all generic state writes or duplicating runtime state. SQLite busy contention remains a handled store outcome; short local transactions contain no network calls. Additional table/index/storage size and retained receipts are accepted design costs, not measured capacity or a new service requirement.

For design-only rollback, supersede this ADR in a reviewed documentation commit. Once implemented, disable only the new release entry point while retaining basic Admin/B1 and local applied state. Preserve CT release/journal and Runtime receipts during source rollback. Never drop the new tables, reset revision or restore an old database as an automatic code-revert side effect.

Technical references used for transaction reasoning (not proof of a live configuration):

MySQL 8.4 manual, innodb-consistent-read.html and innodb-transaction-isolation-levels.html at https://dev.mysql.com/doc/refman/8.4/en/: consistent snapshot reads and conditional DML observe different views; a snapshot alone is not a concurrency claim.
SQLite transaction reference, https://www.sqlite.org/lang_transaction.html: short BEGIN IMMEDIATE transactions serialize writes and may report SQLITE_BUSY.

Source: new-system/docs/decisions/0028-persistent-release-and-runtime-receipt-contract.md — ctc repository · This page is an English translation of the source document. Identifiers, file paths and code symbols are unchanged.