Skip to content

Incident Hotfix Runbook

Purpose

Use this runbook for production outages, production errors, or severe degraded service. During an incident, the priority is reducing downtime and restoring the affected workflow. Full root-cause analysis, broad regression, refactor, and prevention work happen after mitigation.

This runbook does not authorize production DB writes, Git actions, or deploys by itself. Those actions still require explicit user confirmation.

Incident Principles

  • Mitigation first, root cause second.
  • Prefer rollback when it is the fastest safe recovery.
  • Keep fixes minimal, reversible, and limited to the failing surface.
  • Do not bundle cleanup, wording changes, restrictions, or adjacent improvements.
  • Preserve visible errors. Do not hide failures with empty states or silent success.
  • Record enough evidence to support post-incident RCA after recovery.

Triage

Classify the incident quickly:

Question Examples
Impact full site down, login down, one page down, one API failing, one tenant/user role affected
First failing surface URL, API route, button/action, scheduled job, integration
Failure mode 500/fatal, 403, bad JSON, timeout, incorrect data, missing config, DB error
Recent change latest deploy, config change, data repair, cron/integration run
Fastest safe mitigation rollback, config restore, minimal code fix, limited data correction

Use logs and production observations to narrow the failing surface. Do not spend incident time proving a full root cause if rollback or a minimal fix is already clear.

Mitigation Order

  1. Confirm whether rollback is available and safe.
  2. If rollback is fastest, recommend rollback and verify recovery.
  3. If rollback is unsafe or slower, make the smallest patch that restores the failing workflow.
  4. If data correction is required, get explicit production DB write confirmation and keep the write bounded.
  5. Verify the exact production symptom is resolved.
  6. Stop broad work and move remaining cleanup to RCA/follow-up.

Minimal Verification

Run only checks needed to prove recovery:

  • failing URL/API now returns the expected status and response shape;
  • affected user workflow reaches the expected visible state;
  • recent logs no longer show the same fatal/500/error spike;
  • if data was repaired, the affected row/state is correct and no wider mutation was made.

Do not block recovery on full field-process regression unless the mitigation itself changes a broad field workflow and no rollback is available.

Hotfix Scope Guard

Before editing, state:

Incident scope:
- Restore:
- Out of scope until RCA:
- Must preserve:
- Rollback option:
- Minimal verification:

Allowed during incident:

  • rollback;
  • narrow config restore;
  • one-purpose code patch;
  • bounded production data correction with explicit approval;
  • one focused production verification path.

Not allowed during incident unless explicitly requested:

  • refactor;
  • broad test suite expansion;
  • wording/UI copy cleanup;
  • new restrictions or permission model changes;
  • adjacent bug fixes;
  • schema redesign;
  • large business logic rewrite.

Post-Incident RCA

After service is restored, create or connect follow-up tracking when needed. The RCA should cover:

  • timeline and impact;
  • triggering change or condition;
  • why existing tests/review/release gate missed it;
  • permanent fix;
  • unit/integration/browser test to prevent recurrence;
  • runbook or alerting improvement;
  • whether rollback/canary process needs adjustment.

RCA work follows the normal development process and should not remain mixed into the emergency hotfix session unless the user explicitly asks for immediate follow-up.

---

Source: docs/runbooks/INCIDENT_HOTFIX_RUNBOOK.md — ctc repository · This page is an English translation of the source document. Identifiers, file paths and code symbols are unchanged.