Skip to content

Platform Deployment Health, Rollback And Restore Runbook

  • Status: SOURCE-RECONCILED ADMIN/OPERATIONS DRAFT / DOCUMENTATION ONLY
  • Source baseline: qa/master@8e6614514b6e330f7efca60da68a162bb1de50db
  • Audience: Release Owner, Deployment Operator, Infrastructure/DB/File Owner, Integration Owner, Support Operator, Incident Owner and approved Process Canary Owner
  • Source authority: PLATFORM_DEPLOYMENT_ENVIRONMENT_OBSERVABILITY_RECOVERY_SOURCE_AUTHORITY.md
  • Rebuild packet: PLATFORM_DEPLOYMENT_ENVIRONMENT_OBSERVABILITY_RECOVERY_REBUILD_PACKET.md
  • Current release procedure: RELEASE_GATE_RUNBOOK.md
  • Current production environment: PRODUCTION_ENVIRONMENT_RUNBOOK.md
  • Current effect: repository documentation only; no Git promotion, server inspection, service operation, backup, restore, provider, credential, Wiki or Production mutation

1. Purpose And Boundary

Use this Runbook for an explicitly authorized deployment-health investigation, rollback decision, backup review, restore rehearsal or disaster-recovery operation.

Do not use it to add extra mandatory steps to every ordinary CTC release. The canonical normal deployment boundary remains the Release Gate Runbook.

This Runbook is required when the question is broader than remote Git promotion, for example:

  • “Did Production actually load the intended release?”
  • “The site responds, but a workflow is failing.”
  • “Which PHP/Node/config/schema version is running?”
  • “A scheduler or callback appears stale.”
  • “The Voice Gateway is down or transcripts are backlogged.”
  • “Logs are missing or the log filesystem may be full.”
  • “File metadata exists but the object cannot be read.”
  • “Can this backup actually restore the whole system?”
  • “What must remain disabled after a database restore?”
  • “Can we fail over or fail back safely?”

1.1 Separate completion states

Git promoted
!= source applied to server
!= process restarted/reloaded
!= service live
!= service ready
!= schema compatible
!= provider ready
!= business canary accepted
!= stabilized

Report the highest level actually proven.

2. Authorization Checklist

Before any external action, record authorization separately.

Effect Examples Separate authorization required
Git promotion qa -> master yes, standard deployment instruction
Server read SSH, service status, installed config, deployed SHA yes when not already included in the authorized investigation
Service change restart/reload/enable/disable/drain yes
DB read identity/schema/backup/restore inspection yes, environment-specific
DB write/restore migration, snapshot restore, failover explicit exact operation
File/object read object inventory/readability environment/storage authority
File/object write restore/delete/copy explicit exact operation
Config/credential env/service/OAuth/token change explicit exact operation
Provider QBO/Gmail/Telegram/Twilio/Gemini action explicit effect
DNS/callback routing failover/failback explicit effect
Wiki publication T_WIKI_PAGE named-slug authorization

If authority is absent, complete documentation/read-only analysis and report the smallest required next effect.

3. Required Investigation Packet

Collect the smallest safe set.

Incident/release/rehearsal identifier
Requested effect and authorization
Environment and deployment ID
Hosting mode: shared/dedicated/multitenant
Expected source SHA/release ID
Expected config release ID
Expected schema release/version
Current remote qa/master
Affected service(s)
Affected tenant/profile/roles/processes
Affected provider/file/scheduler family
Start time and timezone
Visible symptom or release result
Whether user/provider/physical actions were repeated
Known backup set / recovery point
Current traffic/effect state

Do not collect or paste:

  • passwords;
  • private keys;
  • access/refresh tokens;
  • session cookies/IDs;
  • API secrets;
  • full environment files;
  • raw provider authorization headers;
  • unrestricted customer documents/messages;
  • signed object URLs when a safe reference is enough.

4. Fast Decision Tree

Is this only a Git promotion question?
-> use Release Gate Runbook and remote ref readback.

Is the runtime unavailable or wrong?
-> identify service/release/config/schema before restarting or rolling back.

Is the runtime live but workflow failing?
-> check readiness, dependencies and owning process; do not restart blindly.

Is provider/file/physical work involved?
-> stop replay, reconcile owner-specific evidence first.

Is data loss/corruption suspected?
-> contain/freeze, preserve evidence, identify backup/recovery point before writes.

Is a complete restore required?
-> create isolated target, keep traffic/schedulers/providers disabled, follow restore sequence.

5. Deployment Identity Investigation

5.1 Expected identity

Record:

remote qa SHA
remote master SHA
approved release/range
expected config release/reference
expected migration/schema set
expected service definitions

5.2 Observed runtime identity

Where separately authorized, inspect the narrowest evidence available:

  • authenticated page window.CTC_RELEASE_SHA;
  • server worktree HEAD or release environment variable;
  • service process start/restart generation;
  • current config release/reference;
  • current DB identity/schema check;
  • Voice Gateway service/build/health response;
  • installed scheduler/service definitions.

5.3 Classification

Result Meaning Action
MATCHED observed intended identity continue readiness/canary
GIT_ONLY remote master matches, runtime not inspected do not claim runtime applied
STALE_RUNTIME server/service runs older source inspect auto-pull/service apply failure
MIXED_RELEASE service instances/artifacts disagree stop expansion, drain/repair
CONFIG_DRIFT source matches, effective config differs config owner review/reapply/rollback
SCHEMA_DRIFT service expects different schema stop writes; migration/fail-forward owner
UNKNOWN identity unavailable do not claim healthy release

6. Current Production Environment Checks

These are examples for a separately authorized server inspection. They are not automatically run by this documentation change.

6.1 Files and permissions

Confirm only the expected metadata, not secret values.

/etc/ctc/prod.env
/etc/ctc/gmail/credentials.json
/etc/ctc/voice-gateway.env
/var/log/ctc
/var/lib/ctc-ai-voice-gateway
/var/www/html

Check:

  • exists where expected;
  • correct owner/group/mode;
  • runtime user can read/write only what is required;
  • no secret copied into Git worktree;
  • worktree-generated state is not relied upon across git reset --hard;
  • disk/inode capacity is sufficient.

Do not print file contents.

6.2 PHP-FPM

Check when authorized:

  • service active and process generation/start time;
  • environment file drop-in loaded;
  • PHP-FPM config syntax;
  • clear_env contract;
  • runtime release identity;
  • PHP error log for the affected time;
  • local bounded HTTP response;
  • selected DB/schema/readiness check.

A restart is a write effect and must follow authorization. Do not restart only to see whether a symptom disappears.

6.3 Apache

Check:

  • service active;
  • current virtual host/rewrite/proxy config;
  • /api routing;
  • PHP handler availability;
  • Voice Gateway WebSocket proxy when relevant;
  • TLS/network edge ownership;
  • error/access logs for affected time;
  • config syntax before an authorized reload.

6.4 MySQL

First verify environment identity:

hostname
port
database
current user
version

Then run only the read query required for the affected surface.

Do not perform a migration, repair, snapshot restore or failover from this Runbook without exact authorization and rollback/fail-forward ownership.

7. Liveness And Readiness Investigation

7.1 PHP Web/API

Current available evidence may include:

  • Apache/PHP process state;
  • bounded page/login response;
  • authenticated release SHA;
  • selected API response with request ID;
  • DB identity/query;
  • selected file/object read;
  • current logs.

Do not infer:

login.php 200
-> all routes ready

one API 200
-> DB schema/provider/file ready

7.2 AI Voice Gateway

Current process health:

GET /healthz

Then separately determine when authorized:

  • gateway enabled flag;
  • systemd unit installed/active/restart count;
  • intended Node build/source;
  • environment file ownership;
  • loopback port/listener;
  • Apache WebSocket proxy;
  • state/outbox directory readability/writability;
  • signed CTC tool/event endpoint reachability without business replay;
  • Twilio/Gemini configuration presence;
  • callback/recording status backlog;
  • transcript outbox count/age/failures.

Do not place a live call merely to prove generic readiness.

7.3 Readiness result

READY
DEGRADED
NOT_READY
FAILED
UNKNOWN

Record every failed/unknown dimension and whether traffic, writes, callbacks, workers or provider effects remain enabled.

8. Common Deployment Incident Procedures

8.1 Remote master correct, Production appears stale

  1. stop declaring the release complete beyond Git boundary;
  2. record expected/observed SHA;
  3. inspect auto-pull service/timer/log only if authorized;
  4. determine whether fetch/reset failed, was not invoked, is locked or points to another worktree;
  5. verify no untracked runtime state will be destroyed;
  6. apply/retry only the authorized deployment mechanism;
  7. read back worktree and runtime identity;
  8. do not copy files manually as a substitute.

8.2 PHP is live but DB unavailable

  1. do not restart repeatedly;
  2. verify DB environment identity without printing credentials;
  3. inspect connection error and affected deployment scope;
  4. determine network/DNS/credential/DB service state;
  5. keep consequential writes disabled if state is unknown;
  6. restore connectivity or fail over only with DB authority;
  7. verify DB identity/schema/read-only query;
  8. run bounded application canary.

8.3 Wrong or missing environment value

  1. identify exact key owner and expected release/reference;
  2. verify current effective source/fallback without printing secret values;
  3. determine affected services;
  4. prepare previous/current value references and rollback;
  5. update external config only with authorization;
  6. restart/reload required services in controlled order;
  7. verify effective config result, readiness and workflow;
  8. preserve provider objects already created under the old config.

8.4 PHP/Apache error spike after release

  1. stop rollout/expansion;
  2. correlate deployment/source/request IDs;
  3. determine whether errors are global, tenant, role, page/API or process-specific;
  4. inspect selected logs and current business rows;
  5. classify source/config/schema/provider/file cause;
  6. choose source rollback or fail-forward with compatibility evidence;
  7. verify health and bounded canary;
  8. retain error evidence and reopen conditions.

8.5 Logs missing or unwritable

  1. verify application remains operational separately;
  2. inspect log root path, permissions, disk and inode capacity;
  3. identify which channels are missing;
  4. determine whether logs fell back into the worktree;
  5. do not delete evidence broadly to free space;
  6. authorize bounded rotation/archive/cleanup if needed;
  7. verify new writes and collector behavior;
  8. record the evidence gap explicitly.

8.6 File metadata exists but object cannot be read

  1. stop re-upload/delete attempts;
  2. read T_FILE metadata/link and storage/path identity;
  3. check object generation/readability through owning file adapter;
  4. determine missing object, permission, expired URL, wrong backend or metadata drift;
  5. restore/reconcile only the missing boundary;
  6. verify object, metadata, business link and download;
  7. preserve orphan/missing-object evidence.

8.7 Scheduler appears stale

  1. identify scheduler definition and expected schedule/timezone;
  2. verify installed/enabled state if authorized;
  3. inspect last invocation and child results;
  4. determine disabled, failed, overlapping, stuck or never installed;
  5. reconcile any provider/local work before manual replay;
  6. trigger only the owning job with explicit authorization;
  7. verify item-level results and next scheduled run.

8.8 Callback failures

  1. identify provider, endpoint, deployment and provider event/object ID;
  2. verify routing/TLS/signature/secret scope without exposing secret;
  3. determine receipt, duplicate, apply and local state;
  4. do not replay provider create/send;
  5. reapply or reconcile callback through the provider owner;
  6. verify local authoritative result and callback backlog.

9. Rollback Decision Procedure

9.1 Freeze expansion

  • stop new rollout cohorts;
  • drain new worker claims if relevant;
  • preserve callbacks and provider evidence;
  • record current release/config/schema/data state.

9.2 Classify changed components

Source
Config
Schema
Business data
File objects/metadata
Provider objects/callbacks
Scheduler/worker state
Physical operations

9.3 Choose action

Condition Action
previous source compatible with current schema/data source/config rollback possible
schema/data changed incompatibly fail-forward or data restore owner required
provider object already accepted reconcile/compensate; do not replay/delete generically
file object changed object/link-specific recovery
physical work completed system reconciliation, not physical replay
identity unknown stop and gather evidence

9.4 Verify rollback

  • observed source/config/schema identities;
  • liveness/readiness;
  • selected business rows;
  • provider/file state;
  • target-role canary;
  • residual drift;
  • stabilization/reopen criteria.

10. Backup Inventory Review

Use a component matrix, not a single “backup enabled” answer.

Component Backup owner Current backup/ref Last success Verification Restore test Gap/next owner
Runtime DB
Archive DB/manifests
T_FILE metadata
EFS/S3 objects
External config releases
Credentials/tokens recovery plan, not raw values
Source artifacts
Voice gateway outbox/state
Logs/audit
Control Tower metadata
Provider reconciliation refs

If any mandatory component is missing, classify the set PARTIAL or UNVERIFIED.

11. Isolated Restore Rehearsal

11.1 Preconditions

  • exact backup set selected;
  • isolated target approved;
  • source/config/schema compatibility identified;
  • no public traffic;
  • provider sends disabled;
  • callbacks isolated or safely routed;
  • schedulers/workers disabled;
  • restore owner and cleanup owner assigned;
  • evidence and success criteria defined.

11.2 Execution

  1. create isolated infrastructure;
  2. record start time;
  3. restore DB recovery point;
  4. verify DB identity and restored data time;
  5. restore/reconcile archive state;
  6. restore object generation/inventory;
  7. reconcile T_FILE metadata/link/readability;
  8. apply config references;
  9. recover/rotate/reauthorize credentials through owners;
  10. deploy exact compatible source;
  11. start web/API in restricted mode;
  12. verify release/config/schema/liveness/readiness;
  13. inspect voice gateway/outbox state where applicable;
  14. reconcile provider/callback state;
  15. run read-only canaries;
  16. run authorized transactional canaries and cleanup;
  17. enable one scheduler/provider family if rehearsal scope requires it;
  18. measure completion time and data loss interval;
  19. record failures/unknowns;
  20. clean up the isolated target only after evidence retention approval.

11.3 Required readback

Backup set and recovery point
Restored DB identity/time/schema
File object/metadata/link counts and sampled readability
Config release/effective state
Credential availability or reauthorization result
Source/service identities
Health/readiness dimensions
Provider/callback/outbox reconciliation
Business canary results and cleanup
Measured RPO/RTO
Residual risks

12. Disaster Recovery Procedure

12.1 Declare and contain

  • assign incident owner;
  • record last known release/config/schema and current authoritative data location;
  • stop unsafe writes/effects where possible;
  • preserve logs/audit/provider references;
  • identify whether primary can recover faster than failover.

12.2 Select recovery set

  • choose latest verified set within accepted data-loss bounds;
  • identify missing components;
  • obtain owner decision for known loss/partial recovery;
  • do not claim RPO if not established.

12.3 Restore and reconcile

Follow isolated restore sequence. Keep public traffic and outbound effects blocked until canaries/reconciliation pass.

12.4 Enable traffic and effects

readiness
-> provider/callback reconciliation
-> target-role canary
-> controlled traffic
-> one scheduler/worker family
-> outbound provider effects last
-> observation

12.5 Stabilize

Track:

  • errors/latency;
  • schema/data conflicts;
  • missing files;
  • provider/callback backlog;
  • scheduler/worker lag;
  • user reopen reports;
  • measured recovery times.

13. Failback Procedure

  1. confirm rebuilt primary is ready and compatible;
  2. drain writes/effects on recovery deployment;
  3. capture final change/provider/file manifest;
  4. synchronize/restore primary;
  5. reconcile provider/callback ownership;
  6. verify source/config/schema/service identities;
  7. run business canaries;
  8. shift traffic/callbacks;
  9. enable effects gradually;
  10. observe;
  11. retain evidence and rollback window;
  12. retire recovery deployment only with authorization.

Do not fail back by DNS change alone.

14. Shared/Dedicated Checklist

For each deployment verify:

deployment ID and hosting mode
portable tenant mapping
source/config/schema release
DB/object/credential references
service and scheduler inventory
health/readiness contract
logs/metrics/alerts owner
backup component coverage
restore rehearsal status
provider callback routing
rollback/failback
support contacts

Dedicated deployments must not depend on Shared server paths, Client 1 defaults or local integer tenant identity.

15. Escalation Packet

Include:

incident/release/restore ID
environment/deployment/hosting mode
expected and observed source SHA
expected and observed config/schema references
affected service/tenant/profile/role/process
time window/timezone
health/readiness/component state
request/process/provider/scheduler/backup/restore refs
DB/file/provider/callback state
last safe boundary
repeated actions, if any
rollback/fail-forward options
current traffic/effect/scheduler state
next required authority

Exclude secrets and raw customer/provider payloads.

16. Completion Report Template

Operation:
Authorization:
Environment/deployment:
Expected release/config/schema:
Observed release/config/schema:

Services:
- Web/API:
- Worker/schedulers:
- Voice Gateway:

Dependencies:
- DB:
- Files:
- Config/credentials:
- Providers/callbacks:

Observability:
- Logs:
- Metrics/traces:
- Alerts:

Backup/restore:
- Backup set:
- Components:
- Restore target/result:
- Measured RPO/RTO:

Business canary:
- Tenant/role/process/scenario:
- Readback/cleanup:

Result level:
- Git promoted / Runtime applied / Ready / Field accepted / Stabilized

Residual risk:
Rollback/fail-forward:
Next owner/action:

17. Current Evidence Limit

This repository Runbook does not prove current Production service state, deployed SHA, backup settings, file snapshots, scheduler installation, centralized observability or complete restore capability. Those remain environment-specific evidence requirements.

---

Source: docs/runbooks/PLATFORM_DEPLOYMENT_HEALTH_ROLLBACK_AND_RESTORE_RUNBOOK.md — ctc repository · This page is an English translation of the source document. Identifiers, file paths and code symbols are unchanged.