Skip to content

Platform Deployment, Environment, Observability And Recovery — Source Authority

1. Authority Decision

CTC deployment and recoverability are composed of several independent authorities. They must not be represented by one generic deployed, healthy, backedUp or restored flag.

Source Release Authority
= exact Git commit and reviewed repository range

Deployment Authority
= the named environment/deployment that is intended to run a release

Runtime Instance Authority
= the actual web, API, worker or gateway process instance and its observed release/config identity

Configuration Authority
= the applied non-secret configuration release and credential references

Schema/Data Authority
= the actual database identity, schema state and business data

File/Object Authority
= object storage state plus database metadata/link/readability

Scheduler Authority
= the actual scheduler definition, last invocation and child result

Observability Authority
= request, process, provider, service and deployment evidence

Backup Authority
= a bounded recovery set covering every required persistent component

Restore Authority
= an isolated restore attempt with reconciled component results

Business Acceptance Authority
= the approved target role completing the selected workflow on the intended release

1.1 Required invariants

Remote master moved
!= Production server pulled the commit

Production worktree contains a commit
!= PHP-FPM loaded the expected environment

PHP process is live
!= application is ready

HTTP 200 from login.php
!= authenticated workflow is ready

Voice Gateway /healthz returns 200
!= Twilio, Gemini, CTC tool endpoints, DB or transcript storage are ready

Database connection succeeds
!= schema is compatible
!= selected workflow is correct

Config file exists
!= effective config is valid
!= provider credential is usable

Provider credential is usable
!= provider effect is permitted

Archive copy exists
!= disaster-recovery backup exists

Soft-deleted data
!= backup

Database snapshot exists
!= file objects, config, credentials and provider references are recoverable

Backup completed
!= restore verified

Restore completed
!= reconciliation passed
!= schedulers/providers may be enabled

Source rollback
!= schema/data rollback
!= file rollback
!= provider compensation

A deployment is reachable
!= field accepted
!= stabilized

1.2 No single universal health status

A support or deployment view may produce a summary, but it must retain the underlying dimensions.

LIVE
READY
DEGRADED
NOT_READY
FAILED
UNKNOWN

The summary must identify which dependency or acceptance dimension produced the result. UNKNOWN must not be displayed as healthy.

2. Current Source-Reconciled Runtime Topology

2.1 Current primary PHP application

Current source and runbooks establish the following intended production topology.

Internet / External TLS
        |
        v
Apache HTTP Server
        |
        +--> PHP page request -> index.php -> PHP-FPM
        |
        +--> /api/* -> api/index.php -> PHP-FPM
        |
        +--> public provider callback / OAuth / cron endpoints -> PHP-FPM
        |
        +--> /voice/twilio-media -> loopback WebSocket proxy -> AI Voice Gateway

PHP-FPM
        |
        +--> MySQL / RDS-compatible endpoint
        +--> EFS-style media path or S3 object storage
        +--> /var/log/ctc or repository fallback logs
        +--> Gmail / QBO / QuickBooks Time / Telegram / Twilio / Gemini
        +--> optional Control Tower database

Current confirmed source characteristics:

  • Apache rewrite routes /api to api/index.php and user pages to index.php?page=<slug>.
  • The current application is server-rendered PHP with Bootstrap compatibility and newer Tailwind surfaces.
  • PHP-FPM receives production environment variables from an external environment file according to the production runbook.
  • MySQL access is synchronous PDO through KJDatabase.
  • DEPLOYMENT_TYPE currently defaults to saas in source.
  • integer CLIENT_ID remains deployment-local compatibility identity.
  • application file storage supports EFS-compatible local paths or S3 object storage.
  • provider callbacks and public integration endpoints are served by the same Apache/PHP deployment.

Current runtime topology diagram

flowchart TB
    USER[Browser / Mobile User]
    PROVIDER[Provider Callback / OAuth / Webhook]
    TLS[External TLS / Network Edge]
    APACHE[Apache]
    PHP[PHP-FPM Application]
    DB[(MySQL Runtime DB)]
    ARCH[(prod_archive where configured)]
    FILES[(EFS-compatible Media or S3)]
    LOGS[(JSONL / Service Logs)]
    CT[(Optional Control Tower DB)]
    EXT[External Providers]
    VG[AI Voice Gateway Node Service]
    STATE[(Gateway State / Transcript Outbox)]

    USER --> TLS --> APACHE
    PROVIDER --> TLS
    APACHE --> PHP
    APACHE -->|WebSocket proxy| VG
    PHP --> DB
    PHP --> ARCH
    PHP --> FILES
    PHP --> LOGS
    PHP --> CT
    PHP --> EXT
    VG --> EXT
    VG -->|signed tool/event calls| APACHE
    VG --> STATE

2.2 Current production filesystem and environment contract

The production runbook documents these intended external paths.

Application worktree:
/var/www/html

PHP environment:
/etc/ctc/prod.env

PHP-FPM systemd drop-in:
/etc/systemd/system/php-fpm.service.d/override.conf

Gmail OAuth client credentials:
/etc/ctc/gmail/credentials.json

Preferred runtime logs:
/var/log/ctc

AI Voice Gateway environment:
/etc/ctc/voice-gateway.env

AI Voice Gateway persistent state directory:
/var/lib/ctc-ai-voice-gateway

The repository does not prove those paths currently exist, have the documented ownership/mode, or contain the current approved values. That requires separately authorized environment inspection.

2.3 Current source release identity

getReleaseSha() resolves release identity in this order:

CTC_RELEASE_SHA
-> GITHUB_SHA
-> SOURCE_VERSION
-> Git worktree HEAD/ref/packed-ref

The authenticated PHP shell exposes this value to the browser as:

window.CTC_RELEASE_SHA

This is useful runtime evidence, but it has limits:

  • it can be blank when the deployment has no usable environment variable or Git metadata;
  • it identifies source, not config release, schema version, file state or service generation;
  • the browser value requires a successfully rendered authenticated page;
  • it does not prove every service instance uses the same release;
  • it does not prove the automatic pull completed atomically.

2.4 Current Git deployment boundary

Current canonical repository policy:

approved qa candidate
-> merge/push exact approved range to master
-> Production automatic pull is triggered
-> remote master readback proves Git deployment boundary

The repository does not contain the exact production auto-pull service/timer definition, cadence, lock, atomicity, restart behavior or failure ledger.

The production runbook records a possible implementation:

git fetch --prune origin
git reset --hard origin/master

Current source-level conclusion:

master push deployment contract:
CURRENT CANONICAL POLICY

exact automatic pull implementation:
NOT ESTABLISHED IN REPOSITORY

current server deployed SHA:
NOT VERIFIED IN THIS DOCUMENTATION WAVE

2.5 Current PHP service environment

The production runbook documents:

php-fpm EnvironmentFile=/etc/ctc/prod.env
clear_env = no

A configuration update requires PHP-FPM restart before new environment values are visible to PHP workers.

Current gaps:

  • no source-controlled immutable config release manifest;
  • no general effective-config version shown with release SHA;
  • no documented pre-start validation for every required application/provider setting;
  • no current source proof that all production instances share the same environment file;
  • no current config drift query across Shared/Dedicated deployments;
  • no general secret rotation/readback evidence contract.

3. Current AI Voice Gateway Topology

3.1 Service process

The repository contains a systemd unit for the AI Voice Gateway.

User/Group: apache
WorkingDirectory: /var/www/html/voice-gateway
EnvironmentFile: /etc/ctc/voice-gateway.env
ExecStart: node dist/src/index.js
Restart: on-failure
StateDirectory: ctc-ai-voice-gateway

Apache proxies the exact public WebSocket path to a loopback listener.

/voice/twilio-media
-> ws://127.0.0.1:8787/voice/twilio-media

The gateway provides:

GET /healthz
-> { ok: true, enabled: <boolean> }

This proves only that the HTTP process answered and reports its enabled flag. It does not prove:

  • public TLS/WebSocket path is reachable;
  • Twilio signature validation succeeds;
  • Gemini WebSocket credentials/model are usable;
  • CTC tool/event endpoints are reachable;
  • the shared tool secret matches PHP;
  • MySQL is reachable through PHP tool endpoints;
  • transcript outbox directory is writable;
  • transcript drain is succeeding;
  • provider callbacks are reaching PHP;
  • the deployed Node build matches the PHP release.

3.2 Gateway state and filesystem outbox

Current provider-specific persistent state includes:

/var/lib/ctc-ai-voice-gateway/transcript-outbox

The outbox is host-local filesystem state. It is not established as:

  • a multi-instance queue;
  • shared durable storage;
  • a backed-up component;
  • a replicated component;
  • a failover-safe component.

A gateway host replacement without explicit outbox reconciliation can lose pending transcript segments.

3.3 Retention timer

The repository contains a systemd timer and oneshot service.

ctc-ai-voice-retention.timer
OnCalendar=*-*-* 03:15:00 UTC
Persistent=true

ctc-ai-voice-retention.service
ExecStart=php scripts/ai_voice_retention.php

Repository presence does not prove:

  • the unit files are installed;
  • the timer is enabled;
  • the last run succeeded;
  • the provider recording delete completed;
  • failed deletion backlog is within limits;
  • the service uses the intended release and environment.

4. Current Scheduler And Long-Running Work Inventory

4.1 Repository-observed scheduler forms

Systemd timer:
AI Voice retention

HTTP secret-protected cron endpoint:
Telegram break-expiry processing

CLI entrypoint:
cron.php for selected Gmail/appointment/report work

Provider push/callback:
Gmail Pub/Sub history watch
Twilio call/recording/media callbacks
Telegram webhook
QuickBooks OAuth callbacks

In-process timer:
AI Voice Gateway transcript-outbox drain every 30 seconds

4.2 Current scheduler authority gap

The repository does not establish one scheduler inventory with:

  • scheduler ID;
  • environment/deployment binding;
  • exact schedule/timezone;
  • command/endpoint and release SHA;
  • credential owner;
  • last start/success/failure;
  • child counts and residual items;
  • timeout/overlap/lease policy;
  • next run;
  • alert owner;
  • disable/drain order;
  • restore enablement order.

An HTTP endpoint, CLI file, service unit or timer file in Git does not prove the job is installed or running.

5. Current Observability Authority

5.1 JSONL and local filesystem evidence

Current diagnostic channels include:

  • ajax request/mutation/error evidence;
  • page-api page-attributed API request samples;
  • page-duration summary/sample channels;
  • performance phase timings;
  • provider-specific channels such as Gmail, QuickBooks Time and operational jobs;
  • PHP error log;
  • systemd/journald output for the gateway and services.

Preferred production JSONL root:

/var/log/ctc

Fallback:

<worktree>/storage/logs

5.2 Current telemetry behavior

Current source confirms:

  • request IDs and page-view IDs exist for selected HTTP telemetry;
  • sensitive-looking fields are masked in API logging;
  • write/error/slow requests are retained by policy;
  • successful fast GET telemetry can be sampled by environment flag;
  • page duration is recorded at shutdown;
  • telemetry failures are intended to remain non-fatal to business responses;
  • file-based writers use locks and can contend;
  • the annual page report legacy mode rewrites a whole JSON file;
  • CTC_LOG_ROOT storage type and current performance are not established by source.

5.3 Current missing observability contracts

No repository source establishes a complete current production contract for:

  • centralized log collection;
  • log rotation and storage-capacity alerts;
  • service metrics backend;
  • distributed tracing;
  • application-wide liveness endpoint;
  • application-wide readiness endpoint;
  • dependency health endpoint;
  • deployment/config/schema drift metrics;
  • scheduler last-success metrics;
  • backup age/failure metrics;
  • restore-rehearsal age;
  • formal SLOs/SLIs;
  • alert thresholds, paging routes and acknowledgement owner;
  • service instance inventory;
  • one support timeline spanning PHP, Node, DB, files and providers.

Current source therefore supports diagnostic evidence, not a claim of complete production observability.

6. Current Persistent-State And Recovery Inventory

6.1 Runtime MySQL database

Current application business state is stored in MySQL. Production documentation identifies an RDS-compatible endpoint, but this repository does not establish current external backup settings.

Not established in repository/current documentation wave:

  • automated snapshot policy;
  • point-in-time recovery window;
  • snapshot encryption/key ownership;
  • cross-region or cross-account copy;
  • backup retention;
  • last successful backup;
  • last isolated restore;
  • measured RPO/RTO;
  • DB failover procedure;
  • replica topology;
  • backup alert owner.

6.2 prod_archive

The archive system moves selected terminal business families from prod to prod_archive through guarded copy/readback/delete and supports family restore.

prod_archive
= retention/archive store
!= full database backup
!= point-in-time recovery
!= disaster recovery

The archive runbook explicitly does not own backup infrastructure.

6.3 File objects and metadata

Current file state is split across:

T_FILE metadata and business link
+
EFS-compatible file object or S3 object
+
readability/download policy
+
object-cleanup state

A recoverable backup set must keep metadata and object generations consistent. Restoring only the DB can produce missing objects. Restoring only object storage can produce unlinked/orphan objects.

Current external settings not established:

  • EFS backup/snapshot policy;
  • S3 versioning/object-lock/lifecycle policy;
  • cross-region copy;
  • object inventory/checksum manifest;
  • restore-generation selection;
  • encryption key ownership;
  • media restore rehearsal.

6.4 Configuration and credentials

Persistent external configuration includes at least:

/etc/ctc/prod.env
/etc/ctc/gmail/credentials.json
Gmail tenant token store
/etc/ctc/voice-gateway.env
Apache/systemd installed configuration
Control Tower deployment/config metadata where enabled

Credentials and tokens are not ordinary backup payloads. Recovery may require one of:

  • restore from a separately secured secret manager/backup;
  • re-provision from the credential owner;
  • rotate and reauthorize;
  • reconnect provider OAuth;
  • invalidate old credentials.

The current repository does not define one approved credential recovery plan for every provider.

6.5 Logs, audit and support evidence

Logs and transition/domain evidence have different retention and recovery requirements.

  • JSONL logs may be ephemeral diagnostics unless an approved collector/retention policy preserves them.
  • Transition audit is intended as immutable reference evidence where installed and written.
  • domain evidence may reside in business tables, provider state or file objects.
  • provider-side state is not reproduced by restoring local data.

A restore must not manufacture provider acceptance or physical completion from logs.

6.6 Voice gateway state

Recoverable voice state spans:

MySQL AI Voice rows
Twilio provider objects/callbacks/recordings
Gemini session (not durable)
filesystem transcript outbox
transcript rows
operation events
recording retention status

The host-local outbox and provider recording state require reconciliation after host or DB restore.

7. Archive, Backup, Restore And Rollback Separation

Operation Purpose Current owner/evidence Does not imply
Archive move old terminal families to archive DB Data Archive Runbook/script disaster backup or PITR
Source rollback run previous source release Git/release owner DB/file/provider rollback
Config rollback apply previous config values/release environment/config owner source/schema rollback
Schema rollback/fail-forward return schema compatibility migration owner business data recovery
DB backup/restore recover database state external DB infrastructure owner file/provider/config recovery
File backup/restore recover object bytes/generation storage owner T_FILE link correctness
Provider reconciliation connect provider state to local records integration owner local DB rollback
Business compensation reverse an accepted business/provider/physical result process owner infrastructure recovery
Disaster recovery rebuild a viable deployment from a bounded recovery set deployment/DR owner field acceptance without canary

8. Current Shared And Dedicated Deployment State

Source defines:

DEPLOYMENT_TYPE = saas

Control Tower metadata supports environment and hosting mode values such as shared, dedicated and multitenant.

Current source conclusion:

Shared SaaS current source path:
PRESENT

Dedicated current production deployment parity:
NOT ESTABLISHED

One repeatable artifact/config/backup/restore contract across Shared/Dedicated:
TARGET / PLANNED

Dedicated deployment must not copy these compatibility assumptions:

  • hardcoded CLIENT_ID=1;
  • deployment-local integer identity as portable identity;
  • implicit local filesystem state;
  • provider credential ownership by default deployment;
  • one server-specific scheduler as global authority;
  • one unversioned environment file as full configuration authority.

9. Current Health And Readiness Summary

Component Current liveness evidence Current readiness evidence Current gaps
Apache/PHP login/page HTTP response selected page/API/DB behavior only when separately tested no general health/readiness/version endpoint
MySQL successful application or identity query selected schema/table/query checks no source-confirmed backup/failover/readiness contract
File storage object operation/readability per file selected file object + metadata/link reread no global readiness/inventory/backup state
AI Voice Gateway /healthz process response none comprehensive provider/tool/outbox/dependency readiness missing
Provider callbacks individual endpoint response/log domain state/callback apply reread no shared callback lag/last-success view
Scheduled jobs process/endpoint output where invoked child-level readback no consolidated installed/last-success inventory
JSONL logging successful file append log path writable for that write rotation, capacity, collection and alerting unknown
Control Tower optional registry query deployment metadata when DB/schema configured not request-time runtime authority; active state unverified

10. Source Discrepancies And Documentation Risks

DEP-DISCREPANCY-001 — Git deployment versus deployed runtime

Remote master readback proves the Git promotion boundary. It does not prove the automatic pull, service reload, runtime release, schema or workflow.

DEP-DISCREPANCY-002 — Auto-pull implementation not in repository

The repository describes the deployment contract but does not contain the installed timer/service implementation. Current cadence, lock and failures require server evidence.

DEP-DISCREPANCY-003 — No application-wide readiness contract

The PHP application has no source-confirmed general liveness/readiness endpoint. Voice /healthz is insufficient for dependency readiness.

DEP-DISCREPANCY-004 — Archive described alongside recovery work

prod_archive has guarded restore behavior but is not a full backup. Operator documentation must not treat archive completion as DR readiness.

DEP-DISCREPANCY-005 — Backup infrastructure not source-established

No current repository evidence establishes RDS PITR/snapshots, EFS backup, S3 versioning, config/secret backup or tested complete restoration.

DEP-DISCREPANCY-006 — File-based telemetry may be on unknown storage

The source maps file writers and contention, but current filesystem type, rotation, capacity and collectors are unknown.

DEP-DISCREPANCY-007 — Voice gateway host-local state

Transcript outbox state is host-local. Multi-instance/failover/restore semantics are not established.

DEP-DISCREPANCY-008 — Deployment type is not a complete profile

The source constant saas and Control Tower hosting metadata do not by themselves establish Shared/Dedicated operational parity.

DEP-DISCREPANCY-009 — RPO/RTO absent

No approved or measured RPO/RTO was found for the whole system. Do not invent numeric objectives in implementation or marketing.

DEP-DISCREPANCY-010 — Runtime service state not verified

The repository contains systemd/Apache definitions, but installation, enablement, runtime generation and current logs were not inspected in this documentation wave.

11. Current Evidence Ceiling

Repository topology:
SOURCE-RECONCILED

Production runbook/environment contract:
DOCUMENTED

Current server/service/runtime state:
NOT VERIFIED

Current deployed SHA/config/schema:
NOT VERIFIED

Current DB backup/PITR:
NOT VERIFIED

Current file/object backup:
NOT VERIFIED

Current config/credential recovery:
NOT VERIFIED

Current centralized observability/alerts:
NOT ESTABLISHED

Current restore rehearsal for complete system:
NOT ESTABLISHED

Shared/Dedicated parity:
TARGET INPUT ONLY

Phase implementation:
USER HOLD

12. Downstream Consumers

This source authority is an input to:

  • the deployment/environment/observability/recovery rebuild packet;
  • release and production environment runbooks;
  • future P9 packaging/repeatability work when explicitly activated;
  • future P12 React + Next.js service deployment design;
  • future worker deployment design;
  • future P11 database transition and rollback;
  • support, incident and disaster-recovery procedures;
  • Wiki projection for authorized platform operators.

It does not activate those tasks or authorize any external effect.

---

Source: docs/reconstruction/PLATFORM_DEPLOYMENT_ENVIRONMENT_OBSERVABILITY_RECOVERY_SOURCE_AUTHORITY.md — ctc repository · This page is an English translation of the source document. Identifiers, file paths and code symbols are unchanged.