Platform Deployment, Environment, Observability And Recovery — Rebuild Packet¶

Status: DETAILED REBUILD INPUT / DOCUMENTATION ONLY
Current source baseline: qa/master@8e6614514b6e330f7efca60da68a162bb1de50db
Source authority: PLATFORM_DEPLOYMENT_ENVIRONMENT_OBSERVABILITY_RECOVERY_SOURCE_AUTHORITY.md
Platform index: PLATFORM_FOUNDATION_RECONSTRUCTION_INDEX.md
Release authority: RELEASE_GATE_RUNBOOK.md
Current production environment: PRODUCTION_ENVIRONMENT_RUNBOOK.md
Target Runtime: current PHP compatibility, React + Next.js Web/API, framework-independent TypeScript applications, approved separate Node.js workers, MySQL first, PostgreSQL after stabilization
Phase implementation: USER HOLD
Runtime, infrastructure, DB, service, scheduler, backup, restore, credential, provider, Wiki publication and Production effects: NOT AUTHORIZED

1. Capability Outcome¶

A future deployment or recovery team must be able to reconstruct and operate CTC from explicit contracts rather than server memory.

For each deployment the system must answer:

Which exact source release is intended?
Which exact source release is running on each service instance?
Which configuration release and schema version are applied?
Which deployment, tenant mapping and hosting mode are active?
Which services are live?
Which services are ready for traffic?
Which dependencies are unavailable or degraded?
Which schedulers/workers are enabled, draining, stale or failing?
Which logs, metrics, traces and alerts prove the current state?
Which backup set covers DB, files, config and required recovery references?
Has that backup set been restored in isolation?
What RPO and RTO were actually measured?
What must remain disabled until data/provider reconciliation passes?
What is the exact source/config/schema/provider rollback or fail-forward path?
Which target-role canary proves that the restored/released system delivers the intended operation?

1.1 Non-goals¶

This packet does not authorize:

a deployment infrastructure rewrite;
a Next.js or worker scaffold;
a generic health endpoint with secret-bearing diagnostics;
a generic backup table;
source, schema, data, file, config or provider rollback;
automatic failover;
automatic scheduler/provider reactivation after restore;
generic dual-write or multi-writer deployment;
one numeric RPO/RTO without owner approval and rehearsal;
GitHub Actions or a mandatory repository-wide CI gate;
treating prod_archive as disaster backup;
storing secrets in deployment manifests or evidence;
publishing platform operations Wiki content;
modifying the current qa -> master release contract while Phase implementation is held.

2. Target Authority Model¶

flowchart TB
    SRC[Source Release]
    CFG[Config Release]
    SCH[Schema Release]
    DEP[Deployment Manifest]
    WEB[Web Runtime]
    API[API Runtime]
    WORK[Worker Runtime]
    GATE[Voice Gateway]
    DB[(Runtime Database)]
    OBJ[(File/Object Storage)]
    PROV[Providers]
    OBS[Logs / Metrics / Traces / Alerts]
    BAK[Backup Set]
    RST[Restore Attempt]
    CAN[Business Canary]

    SRC --> DEP
    CFG --> DEP
    SCH --> DEP
    DEP --> WEB
    DEP --> API
    DEP --> WORK
    DEP --> GATE
    WEB --> DB
    API --> DB
    WORK --> DB
    WEB --> OBJ
    API --> OBJ
    WORK --> PROV
    GATE --> PROV
    WEB --> OBS
    API --> OBS
    WORK --> OBS
    GATE --> OBS
    DB --> BAK
    OBJ --> BAK
    CFG --> BAK
    DEP --> BAK
    BAK --> RST
    RST --> CAN

Concern	Target owner	Must not own
Source release	Git/release owner	DB/provider state
Deployment manifest	deployment application/control plane	secrets or business state
Runtime liveness/readiness	each service adapter	field acceptance
Config release	config control plane/local applied state	source/schema release
Schema version	database migration owner	provider readiness
Business canary	owning Process/product operator	infrastructure truth outside selected flow
Observability	observability/support applications	business transition authority
Backup set	backup/recovery owner	archive eligibility or provider compensation
Restore attempt	recovery owner	target-role acceptance by itself
Provider reconciliation	provider integration owner	DB/file restore
Failback	incident/deployment owner	deleting evidence or skipping reconciliation

3. Target Deployment Manifest¶

Every deployment or release candidate must be addressable by an immutable manifest or equivalent derived record.

3.1 Logical contract¶

DeploymentManifest
  schemaVersion
  releaseId
  deploymentId
  environment
  hostingMode
  sourceSha
  sourceRepository
  sourceRange / changeSetRef
  configReleaseId
  schemaReleaseId
  applicationArtifactRef
  webArtifactRef
  workerArtifactRefs[]
  serviceVersions[]
  migrationSet[]
  previousReleaseId
  rolloutCohort
  createdAt
  createdBy
  approvalRef
  rollbackRef
  status

3.2 Safe manifest rules¶

The manifest stores references and fingerprints, not secret values.

Allowed:

source SHA;
artifact digest;
configuration release ID;
credential owner reference;
schema migration IDs;
service names and versions;
deployment/environment identity;
tenant cohort;
rollback predecessor;
approval/ticket reference.

Forbidden:

passwords;
API keys;
OAuth access/refresh tokens;
raw environment file;
session IDs/cookies;
private keys;
full customer payloads;
provider authorization headers.

3.3 Deployment status¶

PREPARED
APPROVED
APPLYING
APPLIED
PARTIAL
FAILED
ROLLED_BACK
SUPERSEDED
UNKNOWN

APPLIED means the deployment operation finished according to its component contract. It does not mean ready, field accepted or stabilized.

4. Target Runtime Identity¶

Each independently running service instance must expose or log a bounded identity.

RuntimeIdentity
  deploymentId
  environment
  hostingMode
  serviceName
  serviceInstanceId
  sourceSha
  artifactDigest
  configReleaseId
  schemaCompatibilityVersion
  processStartTime
  buildTime
  runtimeVersion
  instanceGeneration

Examples of independent service names:

ctc-php-web
ctc-next-web
ctc-next-api
ctc-worker-<family>
ctc-ai-voice-gateway
ctc-scheduler-<family>

4.1 Identity rules¶

sourceSha must represent the service artifact, not merely remote master.
configReleaseId must identify applied non-secret configuration.
schemaCompatibilityVersion is the version the service expects, not proof the DB currently matches.
instance ID/generation must distinguish rolling replacements.
a service behind a proxy/load balancer must return or log which instance answered in restricted diagnostics.
public health responses remain bounded and non-secret.
a release is drifted when active service identities disagree outside an approved rollout window.

5. Liveness, Readiness And Business Acceptance¶

5.1 State model¶

UNKNOWN
LIVE
READY
DEGRADED
NOT_READY
FAILED
DRAINING
STOPPED

5.2 Liveness¶

Liveness answers:

Can the process/event loop respond without being restarted?

It does not perform expensive provider operations or mutate state.

Target examples:

GET /health/live

Public response example:

{
  "status": "LIVE",
  "service": "ctc-next-web",
  "sourceSha": "<safe short SHA>"
}

5.3 Readiness¶

Readiness answers:

Should this instance receive the traffic assigned to it?

Readiness may check bounded local dependencies required by the service:

required config release loaded;
schema compatibility probe;
DB connection/query appropriate to the service;
writable required local state directory;
required object-storage adapter configuration;
migration/maintenance/drain state;
required internal service endpoint.

Readiness should not call every external provider on every request.

Target example:

GET /health/ready

5.4 Dependency diagnostics¶

Restricted support endpoint/query:

GetDependencyHealth

Possible dimensions:

DATABASE
OBJECT_STORAGE
CONFIG
CREDENTIAL_REFERENCE
WORKER
SCHEDULER
PROVIDER_CALLBACK
CONTROL_TOWER_SNAPSHOT
LOG_STORAGE

Results:

AVAILABLE
DEGRADED
UNAVAILABLE
UNKNOWN
NOT_REQUIRED

5.5 Business canary¶

A business canary proves one bounded Product flow on the intended release.

BusinessCanaryResult
  canaryId
  deploymentId
  releaseId
  tenant/profile
  role
  processId/version
  scenarioId
  startedAt/completedAt
  result
  authoritativeReadbackRefs[]
  cleanupResult
  acceptedBy

A business canary is separate from liveness/readiness.

READY
!= target-role workflow accepted

Canary PASS
!= every process is healthy

6. Target Health Endpoint And Query Set¶

6.1 Public bounded endpoints¶

Target design may expose:

GET /health/live
GET /health/ready

Public responses must not disclose:

DB host/name/user;
filesystem paths;
credential state details;
provider account IDs;
tenant list;
stack traces;
service secrets;
exact internal network topology.

6.2 Restricted support queries¶

GetDeploymentIdentity
GetRuntimeInstances
GetRuntimeHealth
GetDependencyHealth
GetReleaseDrift
GetSchedulerStatus
GetObservabilityStatus
GetBackupInventory
GetRestoreReadiness
GetRestoreAttempt
GetBusinessCanaryStatus
GetFailbackReadiness

All support queries require explicit actor, tenant/deployment scope and safe redaction.

7. Target Deployment And Recovery Commands¶

These are future semantic contracts, not current implemented endpoints.

PrepareDeploymentRelease
ApproveDeploymentRelease
ApplyDeploymentRelease
DrainRuntimeService
ResumeRuntimeService
RollbackSourceRelease
RollbackConfigRelease
ReconcileDeploymentDrift
RegisterBackupSet
VerifyBackupSet
StartRestoreRehearsal
CompleteRestoreReconciliation
AuthorizeSchedulerActivation
DeactivateSchedulers
InitiateFailover
CompleteFailback

Every command specifies:

actor/capability;
deployment/environment;
expected release/config/schema version;
idempotency/request token;
current component state;
changed effect;
exact readback;
partial result;
rollback or fail-forward;
support evidence;
stop condition.

8. Release And Rollout Contract¶

8.1 Current compatibility release¶

Current policy remains:

qa candidate
-> master
-> automatic production pull
-> remote master readback

This remains the Git deployment boundary until an approved replacement exists.

8.2 Target release sequence¶

sequenceDiagram
    participant Owner as Release Owner
    participant Git as Source Authority
    participant CP as Deployment Control
    participant Host as Runtime Host
    participant Health as Health/Drift Read Model
    participant Canary as Target-Role Canary

    Owner->>Git: Approve exact source range
    Owner->>CP: Prepare release manifest
    CP->>Host: Apply source/config/service definitions
    Host-->>CP: Component dispositions
    CP->>Health: Read service identities/readiness
    Health-->>CP: Release/config/schema drift
    CP->>Canary: Run approved bounded scenario
    Canary-->>CP: Authoritative business readback
    alt accepted
        CP-->>Owner: APPLIED / READY / CANARY PASS
    else failure or unknown
        CP-->>Owner: PARTIAL/FAILED + rollback/fail-forward input
    end

8.3 Rollout cohort¶

Possible target cohorts:

INTERNAL
CANARY_TENANT
CANARY_ROLE
PERCENTAGE
NAMED_DEPLOYMENT
ALL

A cohort is allowed only when request/tenant/provider one-writer rules remain intact.

8.4 Drain order¶

Before replacing or rolling back a stateful service:

stop new claims/long-running work;
mark service DRAINING;
finish or safely release in-flight leases;
persist provider/outbox checkpoints;
verify callback routing during rollout;
remove instance from traffic;
replace/restart;
verify identity/readiness;
resume claims only after reconciliation.

9. Rollback Authority¶

Rollback is component-specific.

9.1 Source rollback¶

previous reviewed source release
-> deployment apply
-> runtime identity/readiness

Source rollback does not revert schema/data/provider/file effects.

9.2 Config rollback¶

previous config release
-> restart/reload required services
-> effective-config readback

Config rollback does not change source or provider objects already created.

9.3 Schema rollback or fail-forward¶

A migration defines:

backward compatibility window;
data preservation;
application versions compatible with each schema state;
forward-only repair when reverse migration is unsafe;
backup/restore prerequisite;
caller-zero before destructive changes.

9.4 Provider compensation¶

Provider effects are reconciled/compensated by the integration owner. Source rollback must not replay or delete provider objects automatically.

9.5 File rollback¶

File recovery distinguishes:

restore metadata/link
restore object generation
restore readability/access policy
reconcile orphan/missing object

9.6 Rollback result¶

ROLLED_BACK
PARTIAL
FAIL_FORWARD_REQUIRED
PROVIDER_RECONCILIATION_REQUIRED
DATA_RESTORE_REQUIRED
UNKNOWN

10. Observability Contract¶

10.1 Correlation chain¶

releaseId / deploymentId / serviceInstanceId
-> requestId / pageViewId
-> commandRequestToken
-> processInstanceId / correlationId
-> auditId
-> providerAttemptId / callbackReceiptId
-> schedulerRunId / workerRunId
-> backupSetId / restoreAttemptId

The chain must be queryable without copying raw sensitive payloads into a central log.

10.2 Log contract¶

Structured logs should include when applicable:

timestamp UTC
service/deployment/environment
sourceSha/configReleaseId
severity
request/process/provider/scheduler identifiers
tenant-safe reference
operation/result/error code
duration
instance ID

Logs must mask or omit:

secrets/tokens;
session IDs/cookies;
authorization headers;
raw customer documents/messages unless separately restricted;
full provider payloads;
passwords/private keys.

10.3 Metrics contract¶

Target metric families:

Web/API¶

request count
latency distribution
4xx/5xx rate
DB connection/transaction failure
PHP fatal/error count
active/queued request pressure
release/config/schema drift

Worker/scheduler¶

last run/start/success/failure
run duration
eligible/processed/skipped/failed/unknown counts
overlap/lease conflict
queue/outbox lag where provider-specific queue exists
dead/manual-review count

Provider¶

attempt count
success/definite failure/unknown
callback lag
provider accepted/local pending
credential unavailable/expired
rate-limit count
reconciliation backlog

File/object¶

object write/read/delete failure
metadata-link partial
object-cleanup backlog
unreadable/missing object
storage capacity/latency where available

Backup/restore¶

last successful backup age
backup verification failure
backup component coverage
last restore rehearsal age
restore duration
measured RPO
reconciliation failures

10.4 Trace contract¶

Distributed tracing is a target option, not current source fact. When adopted:

use safe IDs and attributes;
never store raw SQL parameters/secrets;
preserve parent/child request/process/provider relationships;
separate sampled diagnostics from business audit;
do not make trace delivery part of the business transaction.

10.5 Alert contract¶

Every alert definition must specify:

alertKey
source metric/evidence
severity
scope
threshold/window
owner
first action
suppression/maintenance behavior
escalation
recovery condition
runbook link

No numeric threshold is accepted merely because it appears in this packet. Thresholds require measured baseline and owner approval.

11. Scheduler Contract¶

11.1 Scheduler definition¶

SchedulerDefinition
  schedulerId
  deploymentId
  environment
  tenantScope
  triggerType: SYSTEMD_TIMER / CRON / HTTP / PROVIDER_PUSH / IN_PROCESS
  schedule/timezone
  command/endpoint
  sourceSha/handlerVersion
  credentialOwnerRef
  overlapPolicy
  timeout
  leasePolicy
  enabledState
  owner
  runbookRef

11.2 Scheduler run¶

SchedulerRun
  schedulerRunId
  schedulerId
  scheduledFor
  startedAt/completedAt
  sourceSha/configReleaseId
  status
  eligible/processed/succeeded/failed/skipped/unknown
  first/last child reference
  errorCode
  nextSafeAction

11.3 Status¶

ENABLED
DISABLED
DRAINING
RUNNING
STALE
FAILED
UNKNOWN

An installed timer file is not ENABLED; a process exit code 0 is not every-child success.

12. Backup Set Contract¶

12.1 Backup set¶

BackupSet
  schemaVersion
  backupSetId
  deploymentId
  environment
  sourceReleaseId/sourceSha
  configReleaseId
  schemaReleaseId
  createdAt/completedAt
  dataSnapshotRef
  dataRecoveryWindowRef
  archiveManifestRefs[]
  objectSnapshotRef
  objectInventoryRef
  configReference
  credentialRecoveryPlanRef
  serviceStateRefs[]
  logsAuditManifestRef
  controlTowerMetadataRef
  providerReconciliationPlanRef
  encryptionKeyRef
  checksum/manifestDigest
  retentionClass
  legalHoldState
  verificationState
  restoreState

No secret or raw credential is stored in the backup manifest.

12.2 Component coverage¶

A complete backup inventory evaluates:

Component	Example current source	Recovery requirement
Runtime DB	MySQL `prod`	snapshot/PITR/validated restore
Archive DB	`prod_archive`	preserve manifests and family integrity
File metadata	`T_FILE`	same consistency point as business data where required
File objects	EFS/S3	generation/inventory/checksum restore
Config	external env + tenant config release	versioned safe configuration references
Credentials	external stores/token paths	secure restore or reauthorization plan
Source artifact	Git SHA/artifact digest	reproducible artifact availability
Gateway state	AI Voice DB + transcript outbox	reconcile host-local/provider state
Logs/audit	JSONL/audit/domain evidence	retention/legal/support policy
Control Tower metadata	deployment/tenant/config mapping	restore or reconstruct registry safely
Provider state	QBO/Gmail/Telegram/Twilio/Gemini	provider references and reconciliation, not local backup copy

12.3 Backup status¶

PREPARING
COMPLETE
PARTIAL
FAILED
STALE
UNVERIFIED
VERIFIED
RESTORE_VERIFIED
EXPIRED
UNKNOWN

A snapshot becomes RESTORE_VERIFIED only after an isolated restore and reconciliation.

13. Restore Attempt Contract¶

RestoreAttempt
  restoreAttemptId
  backupSetId
  targetDeploymentId
  targetEnvironment
  isolationMode
  startedAt/completedAt
  sourceReleaseId
  restoredDataTime
  componentResults[]
  schemaCompatibilityResult
  fileReconciliationResult
  config/credentialResult
  schedulerState
  providerReconciliationResult
  businessCanaryResults[]
  cleanupResult
  measuredRpo
  measuredRto
  finalStatus
  acceptedBy

Component result:

SUCCEEDED
FAILED
SKIPPED
NOT_APPLICABLE
UNKNOWN
RECONCILIATION_REQUIRED

14. Restore Sequence¶

14.1 Isolated restore¶

sequenceDiagram
    participant Owner as Recovery Owner
    participant Infra as Infrastructure
    participant DB as Database Restore
    participant Files as Object/File Restore
    participant Cfg as Config/Credentials
    participant App as Web/API Runtime
    participant Sched as Schedulers/Workers
    participant Prov as Provider Reconciliation
    participant Canary as Business Canary

    Owner->>Infra: Create isolated target, block user/provider traffic
    Infra->>DB: Restore selected DB point
    DB-->>Owner: DB identity + restored time
    Owner->>Files: Restore/reconcile object generation
    Files-->>Owner: metadata/object/readability result
    Owner->>Cfg: Apply approved config refs / recover credentials
    Owner->>App: Deploy exact compatible source, schedulers disabled
    App-->>Owner: liveness/readiness/release/schema result
    Owner->>Prov: Reconcile pending callbacks/provider objects
    Prov-->>Owner: safe provider dispositions
    Owner->>Canary: Run bounded target-role scenarios
    Canary-->>Owner: authoritative readback
    alt all required results pass
        Owner->>Sched: Enable approved schedulers/workers/providers
    else failure/unknown
        Owner->>Sched: Keep disabled
        Owner-->>Owner: Repair, fail forward or abandon target
    end

14.2 Required restore order¶

declare incident/rehearsal and freeze the selected source state;
identify exact backup set and recovery point;
create isolated network/deployment target;
keep public traffic, callbacks, schedulers and provider sends disabled;
restore DB and verify DB identity/time/schema;
restore/archive reconciliation as applicable;
restore file objects and reconcile T_FILE metadata/link/readability;
apply versioned non-secret config references;
recover/rotate/re-authorize credentials through their owners;
deploy the compatible source artifact;
start Web/API in restricted mode;
verify liveness, readiness, release/config/schema identity;
reconcile pending provider objects/callbacks and host-local outboxes;
run selected read-only and transactional business canaries with cleanup;
authorize traffic;
enable schedulers/workers/providers in controlled order;
observe and record measured RPO/RTO;
preserve evidence and clean up the rehearsal target when authorized.

14.3 Scheduler/provider enable order¶

Default target order:

Web/API read-only or maintenance mode
-> required local DB/file commands
-> callback receipt/apply where safe
-> provider reconciliation
-> one scheduler/worker family at a time
-> outbound provider effects last

The exact order is process/provider-specific and cannot be universalized blindly.

15. Disaster Recovery And Failback¶

15.1 Incident states¶

DETECTED
CONTAINED
RECOVERY_PREPARING
RESTORING
RECONCILING
CANARY
TRAFFIC_ENABLED
STABILIZING
FAILBACK_PREPARING
CLOSED
ABANDONED

15.2 Failover decision inputs¶

outage scope and current authoritative data location;
expected recovery time of current deployment;
latest verified backup/recovery point;
object/config/credential availability;
provider callback routing;
DNS/TLS/network readiness;
scheduler/worker state;
duplicate provider/physical effect risk;
approved data-loss tolerance;
rollback/failback plan;
incident owner and authority.

15.3 Failback¶

Failback is not “switch DNS back.”

drain new writes/effects on recovery deployment;
capture final data/provider/file state;
reconcile changes since recovery point;
restore/synchronize primary target;
verify source/config/schema parity;
reconcile callbacks/outboxes/providers;
run business canaries;
switch traffic;
observe;
retire temporary target only after evidence retention and approval.

16. Shared SaaS And Dedicated Parity¶

16.1 Required common contract¶

Both hosting modes use:

same source repository/release semantics
same process/application contracts
same migration IDs and compatibility rules
same release manifest schema
same liveness/readiness schema
same observability identifiers
same backup/restore manifest schema
same provider/evidence semantics
same rollback/caller-zero rules

16.2 Deployment-specific values¶

Allowed differences:

deployment ID/environment/region;
runtime URLs;
DB/object-store/secret references;
tenant mapping;
resource sizing;
enabled provider/capability set;
scheduling topology;
backup location/retention per approved contract.

16.3 Prohibited assumptions¶

local CLIENT_ID is portable;
Client 1 defaults apply to Dedicated;
one server-local path exists on every deployment;
one host-local outbox survives failover;
a shared provider credential may be reused;
Control Tower is queried on every request;
shared and dedicated have different business process semantics.

17. React + Next.js And Worker Deployment Target¶

17.1 Target service topology¶

flowchart TB
    EDGE[DNS / TLS / Routing]
    NEXT[React + Next.js Web/API]
    PHP[Legacy PHP Compatibility]
    APP[TypeScript Application/Domain]
    WORK[Approved Node Workers]
    VOICE[AI Voice Gateway]
    DB[(MySQL, PostgreSQL later)]
    OBJ[(Object Storage)]
    PROVIDERS[External Providers]
    OBS[Observability]
    CP[Deployment/Config Control]

    EDGE --> NEXT
    EDGE --> PHP
    EDGE --> VOICE
    NEXT --> APP
    PHP --> APP
    APP --> DB
    APP --> OBJ
    WORK --> APP
    WORK --> PROVIDERS
    VOICE --> PROVIDERS
    VOICE --> APP
    NEXT --> OBS
    PHP --> OBS
    WORK --> OBS
    VOICE --> OBS
    CP --> NEXT
    CP --> PHP
    CP --> WORK
    CP --> VOICE

17.2 Migration constraints¶

one request/use case has one authoritative writer;
Next.js and PHP may read-shadow but do not dual-write;
workers do not own business policy;
provider callbacks route to one authoritative handler per cohort;
release manifest covers PHP, Next.js and workers independently;
health/readiness must expose service drift;
restore rehearsal covers both old/new service state during coexistence;
PHP caller-zero precedes retirement;
PostgreSQL follows React/Next.js/MySQL stabilization.

18. Stable Error Catalog¶

DEPLOYMENT_IDENTITY_UNKNOWN
DEPLOYMENT_NOT_FOUND
DEPLOYMENT_APPLY_PARTIAL
DEPLOYMENT_DRIFT
DEPLOYMENT_DRAIN_TIMEOUT
DEPLOYMENT_ROLLBACK_BLOCKED

RELEASE_SOURCE_MISMATCH
RELEASE_CONFIG_MISMATCH
RELEASE_SCHEMA_MISMATCH
RELEASE_ARTIFACT_MISMATCH

SERVICE_NOT_LIVE
SERVICE_NOT_READY
SERVICE_DEGRADED
SERVICE_RELEASE_MISMATCH
SERVICE_INSTANCE_UNKNOWN

DEPENDENCY_DATABASE_UNAVAILABLE
DEPENDENCY_OBJECT_STORAGE_UNAVAILABLE
DEPENDENCY_CONFIG_INVALID
DEPENDENCY_CREDENTIAL_UNAVAILABLE
DEPENDENCY_INTERNAL_SERVICE_UNAVAILABLE

SCHEDULER_NOT_INSTALLED
SCHEDULER_DISABLED
SCHEDULER_STALE
SCHEDULER_OVERLAP_CONFLICT
SCHEDULER_RESULT_UNKNOWN

OBSERVABILITY_LOG_UNWRITABLE
OBSERVABILITY_LOG_CAPACITY_RISK
OBSERVABILITY_METRIC_STALE
OBSERVABILITY_ALERT_OWNER_MISSING

BACKUP_NOT_FOUND
BACKUP_INCOMPLETE
BACKUP_STALE
BACKUP_UNVERIFIED
BACKUP_COMPONENT_MISSING
BACKUP_CHECKSUM_MISMATCH

RESTORE_TARGET_NOT_ISOLATED
RESTORE_COMPONENT_FAILED
RESTORE_SCHEMA_INCOMPATIBLE
RESTORE_FILE_DB_DIVERGENCE
RESTORE_PROVIDER_RECONCILIATION_REQUIRED
RESTORE_CANARY_FAILED
RESTORE_RESULT_UNKNOWN

RPO_NOT_ESTABLISHED
RTO_NOT_ESTABLISHED
FAILOVER_NOT_AUTHORIZED
FAILBACK_BLOCKED

19. Stable Scenario Families¶

Deployment¶

DEP-SCN-010 exact source release apply
DEP-SCN-020 remote master moved / server stale
DEP-SCN-030 mixed service SHAs
DEP-SCN-040 config release mismatch
DEP-SCN-050 schema version mismatch
DEP-SCN-060 partial service restart
DEP-SCN-070 rollback source only
DEP-SCN-080 rollback requiring data/provider reconciliation
DEP-SCN-090 canary failure after readiness
DEP-SCN-100 Shared/Dedicated manifest parity

Health and observability¶

OBS-SCN-010 process live / DB unavailable
OBS-SCN-020 login 200 / selected API fails
OBS-SCN-030 voice health live / transcript outbox unwritable
OBS-SCN-040 provider callbacks stale
OBS-SCN-050 scheduler file present / timer disabled
OBS-SCN-060 log root unwritable/full
OBS-SCN-070 release drift alert
OBS-SCN-080 sampled telemetry interpretation
OBS-SCN-090 support correlation across PHP/Node/provider

Backup¶

BAK-SCN-010 complete DB snapshot inventory
BAK-SCN-020 DB snapshot without file generation
BAK-SCN-030 file snapshot without matching metadata
BAK-SCN-040 config reference missing
BAK-SCN-050 credential requires reauthorization
BAK-SCN-060 archive manifest present / backup missing
BAK-SCN-070 backup checksum mismatch
BAK-SCN-080 backup expires before verification

Restore and DR¶

RST-SCN-010 isolated full restore
RST-SCN-020 DB restore / object restore partial
RST-SCN-030 provider object newer than restored DB
RST-SCN-040 scheduler remains disabled after restore
RST-SCN-050 callback arrives during restore
RST-SCN-060 business canary cleanup
RST-SCN-070 measured RPO/RTO
DR-SCN-010 primary deployment unavailable
DR-SCN-020 failover with host-local outbox
DR-SCN-030 provider callback reroute
DR-SCN-040 failback with post-recovery writes
DR-SCN-050 abandoned restore target cleanup

Migration¶

MIG-DEP-SCN-010 PHP release identity parity
MIG-DEP-SCN-020 Next.js read-only deployment
MIG-DEP-SCN-030 worker read shadow
MIG-DEP-SCN-040 one-writer cohort rollout
MIG-DEP-SCN-050 callback one-handler cutover
MIG-DEP-SCN-060 rollback without dual write
MIG-DEP-SCN-070 Shared/Dedicated parity
MIG-DEP-SCN-080 PHP caller-zero
MIG-DEP-SCN-090 PostgreSQL restore parity

20. Policy Decisions Required Before Implementation¶

Blocking deployment identity decisions¶

authoritative release manifest owner;
whether current auto-pull remains or is replaced;
how source/config/schema/service versions are published;
safe public health response fields;
maintenance/drain mode contract;
rolling/canary cohort and rollback authority.

Blocking observability decisions¶

log destination/rotation/retention;
metrics/tracing platform;
alert severity/owner/escalation;
required service/dependency metrics;
PII/security redaction review;
availability/latency/error SLOs after measurement.

Blocking backup/recovery decisions¶

authoritative DB backup/PITR owner and retention;
file object backup/versioning owner;
config/credential recovery method;
complete backup-set manifest storage;
approved RPO/RTO targets;
restore-rehearsal cadence;
DR/failover authority;
callback/provider handling during restore;
Shared/Dedicated responsibility split.

No implementation writer may invent these policies from infrastructure convenience.

21. Migration Waves¶

DEP-W0
Current deployment/service/scheduler/backup inventory and environment evidence

DEP-W1
Safe release/runtime identity read model

DEP-W2
Service liveness/readiness contract in read-only mode

DEP-W3
Observability correlation and drift diagnostics

DEP-W4
Backup-set inventory and component coverage manifest

DEP-W5
Isolated restore rehearsal with no Production traffic/effects

DEP-W6
One current PHP deployment reference rollout

DEP-W7
One Next.js read-only service deployment

DEP-W8
One approved worker family deployment with drain/readiness

DEP-W9
Shared/Dedicated parity rehearsal

DEP-W10
One-writer transactional/callback cutover support

DEP-W11
PHP/service caller-zero and retirement

DEP-W12
PostgreSQL backup/restore parity after stabilization

Each wave requires explicit activation and applicable infrastructure/DB/provider authorization.

22. Definition Of Ready¶

A deployment/recovery implementation wave is not ready until:

exact deployment/environment and owner are known;
current service and scheduler inventory is verified;
source/config/schema identity contract is accepted;
current auto-pull/service behavior is characterized;
backup component scope is accepted;
rollback/fail-forward and provider/file effects are separated;
safe health/readiness and redaction policy is accepted;
one-writer/callback routing is explicit;
target-role canary and cleanup are defined;
no held Phase is activated implicitly.

23. Definition Of Complete¶

The family reaches rebuild-grade completion only when:

current deployment topology is environment-verified;
current service/scheduler identities are recorded;
release/config/schema drift can be read safely;
liveness/readiness/dependency contracts are implemented and verified;
logs/metrics/traces/alerts have accepted ownership;
complete backup inventory exists;
a complete isolated restore is rehearsed;
measured RPO/RTO are accepted;
file/provider/callback reconciliation is verified;
Shared/Dedicated parity is demonstrated where applicable;
operator Runbook and Wiki content are reviewed/published;
React/Next.js/worker waves preserve one writer, rollback and caller-zero;
stabilization and reopen rules pass.

Current state remains documentation input, not implementation completion.

---¶

Source: docs/reconstruction/PLATFORM_DEPLOYMENT_ENVIRONMENT_OBSERVABILITY_RECOVERY_REBUILD_PACKET.md — ctc repository · This page is an English translation of the source document. Identifiers, file paths and code symbols are unchanged.