This is an internal working record, not an everyday instruction. It was written while the system was being rebuilt, for whoever maintains it. If you are here to do today's work, use the everyday documents instead: Procedures · Processes · Runbooks · The working day, step by step

Platform Deployment, Environment, Observability And Recovery — Policy And Sequence¶

Status: SOURCE-RECONCILED POLICY INPUT / TARGET DESIGN / DOCUMENTATION ONLY
Source baseline: qa/master@8e6614514b6e330f7efca60da68a162bb1de50db
Source authority: PLATFORM_DEPLOYMENT_ENVIRONMENT_OBSERVABILITY_RECOVERY_SOURCE_AUTHORITY.md
Rebuild packet: PLATFORM_DEPLOYMENT_ENVIRONMENT_OBSERVABILITY_RECOVERY_REBUILD_PACKET.md
Current release authority: RELEASE_GATE_RUNBOOK.md
Phase implementation: USER HOLD
Effects: no deployment, service, DB, file, provider, scheduler, backup, restore or Wiki mutation

1. Policy Vocabulary¶

CURRENT
= current source or canonical runbook fact

TARGET
= reconstruction design input, not implemented

OPEN
= owner decision/evidence required

BLOCKING
= required before the affected implementation/recovery claim

COMPATIBILITY
= current path retained during controlled migration

2. Policy Matrix¶

ID	Topic	Current fact	Recommended target	Status
`DEP-POL-010`	Git deployment boundary	`qa -> master`; master triggers auto-pull	retain until replacement is proven; distinguish Git boundary from runtime apply	CURRENT / BLOCKING IDENTITY
`DEP-POL-020`	Auto-pull implementation	installed mechanism absent from repo	named deployment agent/service with lock, attempt/result and source identity	OPEN
`DEP-POL-030`	Release identity	`getReleaseSha()` from env/Git	immutable release manifest + per-service runtime identity	TARGET
`DEP-POL-040`	Config identity	external env/tenant config, no universal release ID	versioned config release reference and effective readback	TARGET
`DEP-POL-050`	Schema identity	migration files/current DB evidence per task	explicit schema release/compatibility contract	TARGET
`DEP-POL-060`	Liveness	PHP page/voice `/healthz` ad hoc	per-service bounded `/health/live`	TARGET
`DEP-POL-070`	Readiness	selected manual checks only	per-service `/health/ready` plus restricted dependency diagnostics	TARGET
`DEP-POL-080`	Business canary	task/release-specific	process/role/scenario/readback/cleanup contract	TARGET
`DEP-POL-090`	Maintenance/drain	provider/service-specific	service state and safe claim/callback drain	OPEN / BLOCKING WORKER ROLLOUT
`DEP-POL-100`	Observability	local JSONL, PHP/systemd logs, page reports	correlated logs/metrics/traces/alerts with safe IDs	TARGET
`DEP-POL-110`	Alert thresholds	not centrally accepted	measured baseline + owner-approved threshold/window	OPEN
`DEP-POL-120`	Scheduler inventory	definitions distributed; install/state unknown	one read-only inventory of definitions/runs/drift	TARGET
`DEP-POL-130`	DB backup	external current state unknown	encrypted snapshot/PITR contract + restore evidence	OPEN / BLOCKING DR
`DEP-POL-140`	File backup	EFS/S3 supported; backup unknown	metadata/object-consistent snapshot/inventory	OPEN / BLOCKING DR
`DEP-POL-150`	Config backup	external env files/config rows	config release/reference recovery; no secret values in manifest	TARGET
`DEP-POL-160`	Credential recovery	provider-specific external stores	restore or rotate/reauthorize plan per credential owner	OPEN
`DEP-POL-170`	Archive	guarded `prod -> prod_archive`	keep separate from disaster backup	CURRENT
`DEP-POL-180`	RPO/RTO	none accepted/measured	owner-approved targets after rehearsal	OPEN / BLOCKING CLAIMS
`DEP-POL-190`	Restore rehearsal	family archive restore only	complete isolated deployment restore and reconciliation	TARGET
`DEP-POL-200`	Scheduler activation after restore	not unified	enable only after local/provider reconciliation, one family at a time	TARGET
`DEP-POL-210`	Provider state after restore	provider-specific/manual	reconcile provider objects/callbacks before replay	TARGET
`DEP-POL-220`	Shared/Dedicated parity	Shared current source; Dedicated target	same manifests/contracts, deployment-specific resources	TARGET
`DEP-POL-230`	Voice Gateway state	host-local transcript outbox	explicit state/storage/failover contract before multi-instance	OPEN
`DEP-POL-240`	Source rollback	Git release rollback	separate source/config/schema/data/file/provider outcomes	TARGET
`DEP-POL-250`	Failover/failback	not established	incident-authorized, evidence-driven, reconciled flow	TARGET
`DEP-POL-260`	Health security	no application-wide endpoint	public bounded health; restricted dependency details	TARGET
`DEP-POL-270`	PostgreSQL recovery	future	backup/restore parity after Next.js/MySQL stabilization	TARGET / LATER

3. Current Git Deployment Sequence¶

sequenceDiagram
    participant User as User / Release Owner
    participant QA as Remote qa
    participant Master as Remote master
    participant Auto as Production Auto-Pull (implementation not in repo)
    participant Host as Production Worktree
    participant PHP as Apache / PHP-FPM

    User->>QA: Approve/commit reviewed candidate
    User->>Master: Merge exact authorized range
    Master-->>User: Remote master SHA
    Note over User,Master: Git deployment boundary proven
    Master-->>Auto: Master changed
    Auto->>Host: fetch + reset/pull (exact installed mechanism unknown)
    Host-->>Auto: success/failure unknown without server evidence
    Note over Host,PHP: Source update does not inherently reload env/processes
    PHP-->>User: Runtime state requires separate evidence

Current interpretation¶

Remote master readback
= authorized Git deployment complete

Remote master readback
!= deployed worktree SHA
!= PHP-FPM process generation
!= environment/config version
!= schema/data/provider health

4. Current PHP Request Topology¶

sequenceDiagram
    participant U as Browser
    participant A as Apache
    participant P as PHP-FPM
    participant DB as MySQL
    participant F as EFS/S3
    participant L as JSONL Logs

    U->>A: Page or /api request
    alt page
        A->>P: index.php?page=slug
    else API
        A->>P: api/index.php
    end
    P->>DB: synchronous query/transaction
    opt file work
        P->>F: object/readability/link operation
    end
    P-->>U: HTML or JSON
    P-->>L: shutdown/request/page telemetry

Policy:

PHP process liveness and DB readiness are separate.
telemetry failure must remain non-fatal to the business response.
a successful page response does not prove provider/scheduler/file readiness.

5. Current AI Voice Gateway Topology¶

sequenceDiagram
    participant T as Twilio
    participant Edge as TLS / Apache
    participant G as Node Voice Gateway
    participant Gem as Gemini Live
    participant PHP as PHP Tool/Event Endpoints
    participant DB as MySQL
    participant O as Host-Local Transcript Outbox

    T->>Edge: WSS /voice/twilio-media
    Edge->>G: loopback WebSocket proxy
    G->>Gem: Live audio session
    G->>PHP: signed tool/event request
    PHP->>DB: business/session/transcript state
    alt transcript delivery succeeds
        G->>PHP: transcript segment
        PHP-->>G: STORED
    else delivery fails
        G->>O: atomic JSON stage
        loop every 30s
            G->>PHP: retry bounded transcript delivery
        end
    end

Required policy:

Gateway /healthz
= process liveness only

Gateway readiness
= config + internal endpoints + state directory + controlled provider diagnostics

Provider diagnostics must not place live calls merely to prove readiness.

6. Target Release Identity Sequence¶

sequenceDiagram
    participant R as Release Owner
    participant Git as Git Source
    participant Cfg as Config Authority
    participant DB as Schema Authority
    participant CP as Deployment Control
    participant S as Service Instances
    participant H as Health/Drift Query

    R->>Git: Select source SHA/range
    R->>Cfg: Select config release
    R->>DB: Select schema compatibility/migrations
    R->>CP: Prepare immutable manifest
    CP->>S: Apply artifacts/config/service definitions
    S-->>CP: component dispositions + runtime identities
    CP->>H: Compare intended vs observed
    H-->>R: source/config/schema/service drift

A release cannot be summarized as one SHA when PHP, Next.js, workers and gateway can have distinct artifacts.

7. Target Liveness And Readiness¶

flowchart TD
    R[Health Request] --> L{Process responds?}
    L -- no --> F[FAILED]
    L -- yes --> D{Draining?}
    D -- yes --> DR[DRAINING]
    D -- no --> C{Required config loaded?}
    C -- no --> NR[NOT_READY]
    C -- yes --> S{Schema compatible?}
    S -- no --> NR
    S -- yes --> DB{Required local dependency available?}
    DB -- no --> NR
    DB -- yes --> Q{Known degraded optional dependency?}
    Q -- yes --> DG[DEGRADED]
    Q -- no --> RD[READY]

Policies:

public liveness is cheap and bounded;
readiness may check required local dependencies;
provider connectivity is not probed through live side effects;
restricted dependency diagnostics provide safe detail;
readiness state includes release/config/schema identity;
maintenance/drain returns NOT_READY or DRAINING even while live.

8. Target Release And Canary Sequence¶

sequenceDiagram
    participant CP as Deployment Control
    participant Old as Old Instances
    participant New as New Instances
    participant DB as Database
    participant P as Provider/Callbacks
    participant Canary as Business Canary

    CP->>New: Start with expected source/config/schema
    New->>DB: readiness/schema probe
    New-->>CP: READY + identity
    CP->>New: Route bounded canary cohort
    Canary->>New: approved role/process scenario
    New->>DB: command/readback
    opt provider effect required and authorized
        New->>P: provider-specific attempt
        P-->>New: provider result
    end
    Canary-->>CP: business result + cleanup
    alt pass
        CP->>Old: Drain and remove from traffic
        CP->>New: expand traffic
    else fail/unknown
        CP->>New: remove from traffic/drain
        CP->>Old: retain/resume traffic
    end

Canary cleanup must not erase evidence required to diagnose the candidate.

9. Source Rollback Sequence¶

sequenceDiagram
    participant O as Incident/Release Owner
    participant C as Current Release
    participant P as Previous Release
    participant DB as Schema/Data
    participant Prov as Providers
    participant H as Health/Canary

    O->>C: Stop expansion / drain new work
    O->>DB: Determine schema/data compatibility
    O->>Prov: Identify committed/unknown external effects
    alt previous source compatible
        O->>P: Apply previous source/config as authorized
        P->>H: Verify identity/readiness
        H-->>O: health + canary result
    else incompatible
        O-->>O: Fail forward, data restore or compensation required
    end

Rollback does not delete or replay provider objects automatically.

10. Scheduler Drain And Resume¶

sequenceDiagram
    participant O as Operator
    participant S as Scheduler
    participant W as Worker/Job
    participant DB as State
    participant P as Provider

    O->>S: Disable new triggers
    O->>W: Mark draining / stop new claims
    W->>DB: Persist/release leases and checkpoints
    opt provider attempt in flight
        W->>P: Reconcile outcome, do not assume failure
        W->>DB: Persist disposition
    end
    W-->>O: drained or blocked items
    Note over O,S: deployment/restore occurs
    O->>DB: Reconcile local/provider state
    O->>S: Enable one scheduler family
    S->>W: Trigger controlled run
    W-->>O: child-level results

11. Backup Set Creation¶

sequenceDiagram
    participant O as Backup Owner
    participant App as Application/Deployment
    participant DB as DB Backup Owner
    participant Obj as File/Object Backup Owner
    participant Cfg as Config/Credential Owners
    participant Man as Backup Manifest

    O->>App: Capture source/config/schema/deployment identity
    O->>DB: Create/identify DB snapshot or recovery point
    DB-->>O: snapshot/time/checksum reference
    O->>Obj: Create/identify object snapshot/inventory
    Obj-->>O: generation/inventory/checksum reference
    O->>Cfg: Capture config release + credential recovery plan refs
    Cfg-->>O: safe references only
    O->>Man: Register component coverage and digest
    Man-->>O: COMPLETE/PARTIAL/UNVERIFIED

Policy:

BackupSet COMPLETE
requires every mandatory component disposition

BackupSet VERIFIED
requires integrity/readability verification

BackupSet RESTORE_VERIFIED
requires isolated restore and reconciliation

12. Database And File Consistency¶

flowchart LR
    B[Business DB State]
    M[T_FILE Metadata/Links]
    O[Object Generation]
    S[Backup Set]
    R[Restore]
    C[Reconciliation]

    B --> S
    M --> S
    O --> S
    S --> R
    R --> C
    C -->|all linked/readable| OK[RESTORE VERIFIED]
    C -->|missing/orphan/conflict| X[RECONCILIATION REQUIRED]

Policies:

exact snapshot technologies may differ by deployment;
the manifest records consistency assumptions;
DB and object backups taken at different points require reconciliation rules;
T_FILE row restore never proves object readability;
object restore never proves the correct business link;
orphan cleanup remains separately authorized.

13. Isolated Restore Sequence¶

sequenceDiagram
    participant O as Recovery Owner
    participant I as Isolated Infrastructure
    participant DB as Restored DB
    participant F as Restored Files
    participant C as Config/Credentials
    participant A as Application
    participant E as Effects/Schedulers
    participant B as Business Canary

    O->>I: Create isolated target, traffic/effects blocked
    O->>DB: Restore selected recovery point
    DB-->>O: DB identity/restored time/schema
    O->>F: Restore object generation/inventory
    F-->>O: object state
    O->>C: Apply config refs, recover/rotate credentials
    O->>A: Deploy compatible source
    A-->>O: liveness/readiness/release/config/schema
    O->>DB: Reconcile DB/file/archive state
    O->>E: Reconcile callback/outbox/provider state, remain disabled
    O->>B: Run bounded canaries and cleanup
    alt all required checks pass
        O->>E: Enable selected schedulers/providers one family at a time
    else failure/unknown
        O->>E: Keep disabled
    end

14. Partial Restore Decision¶

flowchart TD
    P[Restore component result] --> DB{DB restored?}
    DB -- no --> STOP[STOP / abandon or repair]
    DB -- yes --> FILE{Files reconciled?}
    FILE -- no --> RESTRICT[Restricted mode / file-dependent workflows disabled]
    FILE -- yes --> CFG{Config/credentials valid?}
    CFG -- no --> RESTRICT
    CFG -- yes --> SC{Schema/source compatible?}
    SC -- no --> STOP
    SC -- yes --> PROV{Provider/callback state reconciled?}
    PROV -- no --> RESTRICT
    PROV -- yes --> CAN{Canaries pass?}
    CAN -- no --> STOP
    CAN -- yes --> ENABLE[Controlled traffic/effects enablement]

A partially restored system must not silently expose unsupported workflows.

15. Provider Reconciliation After Restore¶

sequenceDiagram
    participant O as Recovery Owner
    participant L as Restored Local State
    participant P as Provider
    participant I as Integration Owner
    participant A as Audit/Support

    O->>L: List pending/unknown/accepted-local-pending effects
    I->>P: Query provider with strongest safe identity
    P-->>I: found / absent / multiple / unknown
    alt found once
        I->>L: Link/reconcile existing provider object
    else absence confirmed
        I->>L: Mark retry eligibility per provider policy
    else multiple or unknown
        I->>L: Manual review required
    end
    I->>A: Preserve attempt/reconciliation references

Provider create/send is not enabled merely because the DB was restored.

16. Voice Gateway Recovery¶

sequenceDiagram
    participant O as Operator
    participant DB as AI Voice DB State
    participant FS as Transcript Outbox
    participant T as Twilio
    participant G as Gateway
    participant PHP as CTC Tool/Event API

    O->>DB: Read active/unknown calls and recording states
    O->>FS: Inventory pending transcript entries
    O->>T: Reconcile provider call/recording objects where authorized
    O->>G: Start disabled or isolated gateway
    G->>PHP: Verify signed tool/event connectivity without business replay
    G-->>O: liveness/readiness/state-dir result
    O->>FS: Drain only idempotent transcript entries
    O->>DB: Verify transcript sequence and call state
    O->>G: Enable traffic after reconciliation

Host-local outbox must be included or explicitly abandoned with evidence in a host replacement.

17. Disaster Recovery Sequence¶

sequenceDiagram
    participant I as Incident Owner
    participant P as Primary Deployment
    participant B as Backup Inventory
    participant R as Recovery Deployment
    participant Prov as Providers/DNS/Callbacks
    participant C as Canary

    I->>P: Contain, freeze, collect last known identities
    I->>B: Select latest verified recovery set
    B-->>I: recovery point/component coverage
    I->>R: Restore isolated deployment
    R-->>I: component/readiness/reconciliation results
    I->>Prov: Prepare callback/DNS routing; outbound effects still blocked
    I->>C: Run approved canaries
    alt accepted
        I->>Prov: Shift traffic/callbacks
        I->>R: Enable schedulers/effects gradually
    else rejected
        I->>R: Keep isolated / repair / abandon
    end

18. Failback Sequence¶

sequenceDiagram
    participant I as Incident Owner
    participant R as Recovery Deployment
    participant P as Rebuilt Primary
    participant D as Data/File Sync
    participant Prov as Providers/Callbacks
    participant C as Canary

    I->>R: Drain new writes/effects
    R->>D: Capture final state and reconciliation manifest
    D->>P: Restore/synchronize primary
    P-->>I: release/config/schema/readiness
    I->>Prov: Reconcile callback and provider ownership
    I->>C: Run primary canaries
    alt pass
        I->>Prov: Shift traffic/callbacks to primary
        I->>P: Enable effects
        I->>R: Observe then retire when authorized
    else fail
        I->>R: Resume recovery deployment if safe
    end

19. Shared SaaS And Dedicated Deployment¶

flowchart TB
    SRC[Common Source Release]
    MIG[Common Migration Contracts]
    MAN[Common Manifest Schema]
    SH[Shared SaaS Deployment]
    DD[Dedicated Deployment]
    SCFG[Shared Config/Credential Refs]
    DCFG[Dedicated Config/Credential Refs]
    SDB[(Shared DB/Object Resources)]
    DDB[(Dedicated DB/Object Resources)]

    SRC --> SH
    SRC --> DD
    MIG --> SH
    MIG --> DD
    MAN --> SH
    MAN --> DD
    SCFG --> SH --> SDB
    DCFG --> DD --> DDB

Policies:

same Product/process/interface semantics;
deployment-local integer IDs are never portable identity;
each deployment owns its backup/restore evidence;
credential and provider ownership are explicit;
dedicated runtime may use different resources but not customer-specific code forks;
caller-zero and rollback are demonstrated independently.

20. Observability Correlation¶

flowchart LR
    DEP[deploymentId/releaseId]
    SVC[serviceInstanceId]
    REQ[requestId/pageViewId]
    CMD[commandRequestToken]
    PROC[processInstanceId/correlationId]
    AUD[auditId]
    PROV[providerAttempt/callback]
    JOB[schedulerRun/workerRun]
    RST[backupSet/restoreAttempt]

    DEP --> SVC --> REQ --> CMD --> PROC --> AUD --> PROV
    SVC --> JOB
    DEP --> RST
    PROC --> RST

A support timeline may connect these references but does not become their write authority.

21. Alert Decision Flow¶

flowchart TD
    S[Signal] --> V{Valid and current?}
    V -- no --> IGN[Ignore/repair observability]
    V -- yes --> M{Maintenance/drain expected?}
    M -- yes --> SUP[Suppress with owner and expiry]
    M -- no --> SEV{Severity and blast radius}
    SEV --> OWN[Notify accountable owner]
    OWN --> RB[Runbook first action]
    RB --> REC{Recovered?}
    REC -- yes --> CLOSE[Record recovery condition]
    REC -- no --> ESC[Escalate / incident]

No alert is complete without owner, first action, recovery condition and runbook.

22. RPO/RTO Policy¶

Current state:

Approved RPO:
NOT ESTABLISHED

Approved RTO:
NOT ESTABLISHED

Measured complete-system RPO/RTO:
NOT ESTABLISHED

Required process:

classify critical business capabilities and persistent components;
establish acceptable data loss and outage with Product/Operations owners;
confirm infrastructure can support the target;
run isolated restore rehearsal;
measure actual restored data time and operator duration;
accept or revise targets;
repeat on a defined cadence.

Do not publish aspirational numbers as achieved recovery capability.

23. Documentation Update Triggers¶

Update the owning source authority, rebuild packet, diagrams, Runbook and Wiki projection when any of these change:

deployment mechanism or auto-pull service;
production environment path/service user;
source/config/schema identity;
PHP/Next.js/worker/gateway topology;
health/readiness endpoint or meaning;
scheduler definition/install/ownership;
log/metric/trace/alert platform;
DB/file/config/credential backup scope;
restore/failover/failback procedure;
RPO/RTO target or rehearsal evidence;
Shared/Dedicated topology;
provider callback routing;
migration/rollback/caller-zero;
business canary or field acceptance.

24. Current Hold Boundary¶

This document does not authorize:

inspecting or changing Production services;
installing systemd/Apache units;
changing auto-pull;
adding health endpoints;
adding metrics/tracing/alerts;
creating backups/snapshots;
restoring DB/files/config;
rotating credentials;
changing callback/DNS routing;
starting/stopping schedulers/workers;
activating P9/P11/P12;
publishing Wiki content;
deployment or Production changes.

---¶

Source: docs/processes/PLATFORM_DEPLOYMENT_ENVIRONMENT_OBSERVABILITY_RECOVERY_POLICY_AND_SEQUENCE.md — ctc repository · This page is an English translation of the source document. Identifiers, file paths and code symbols are unchanged.