Skip to content

Platform Integration And Provider Recovery Runbook

  • Status: SOURCE-RECONCILED ADMIN/OPERATOR DRAFT / DOCUMENTATION ONLY
  • Source baseline: qa/master@a5280b7cdc1c78b809ed6f1996ca31491b522146
  • Audience: authorized Process Owner, Billing/Finance operator, Office/Dispatch, Integration Owner, Support Operator, Deployment Operator and Provider Credential Owner
  • Source authority: PLATFORM_INTEGRATION_OUTBOX_WORKER_PROVIDER_SOURCE_AUTHORITY.md
  • Rebuild packet: PLATFORM_INTEGRATION_OUTBOX_WORKER_PROVIDER_REBUILD_PACKET.md
  • Current effect: repository documentation only; no provider call, retry, worker, DB, credential, Wiki or deployment mutation
  • Important: there is no current universal provider-recovery screen or universal outbox. Use the owning provider/process tools and exact reread.

1. When To Use This Runbook

Use this Runbook when:

  • a provider action timed out;
  • the browser/job returned an error after a local save;
  • a provider may already have created an invoice, sent a message or started a call;
  • a callback arrived but local state did not change;
  • a worker/cron may have stopped mid-run;
  • a file object and metadata/link state disagree;
  • Telegram says suppressed or failed;
  • Gmail history sync needs replay;
  • QuickBooks Time reports a conflicting active timesheet;
  • AI Voice is in PROVIDER_UNKNOWN or recording cleanup failed;
  • the operator does not know whether Retry is safe.

The objective is to identify the last authoritative boundary and continue only the missing owner step.

2. Universal Safety Rules

1. Stop repeating the action.
2. Identify the exact tenant, environment and business source.
3. Read the local business state.
4. Read provider/effect-specific state.
5. Read provider/callback evidence if available.
6. Classify the result.
7. Resume only the missing owner step.

Never infer:

HTTP 200 -> provider delivered
HTTP 500 -> provider failed
Timeout -> provider did nothing
SYNCING -> provider is processing
ERROR -> provider object does not exist
suppressed -> delivered
Webhook accepted -> business transition applied
File row -> object readable
Object deleted -> all links removed
Cron returned -> every child processed

3. Required Investigation Packet

Collect the smallest safe set.

Tenant/client
Environment and deployment/server
Source SHA/release
Physical actor and effective subject
Role/capability
Business source type and ID/UUID
Provider/effect type
Action time and timezone
Request ID
Command request token/request key if present
Process/correlation/audit ID if present
Provider attempt/reference if present
Provider object/message/thread/call/invoice/recording ID if present
Callback event/message/update ID if present
Local state before/after
Visible error/result
Whether the action was repeated

Do not collect or paste:

  • access/refresh tokens;
  • passwords;
  • session cookies/IDs;
  • API keys;
  • Twilio/Gmail/QBO/Telegram secrets;
  • Authorization headers;
  • raw signed callback body unless a restricted incident process requires it;
  • full customer message/document body when references are sufficient.

4. Common Result Classification

Classification Meaning Safe action
NOT_REQUIRED policy says no effect was needed no provider action
SUPPRESSED effect intentionally disabled/skipped communicate through approved fallback if required
READY/PENDING local intent exists, no attempt proven execute only through owning workflow
IN_FLIGHT provider request/callback currently pending wait/observe; do not replay
SUCCEEDED provider result and required local reference exist no repeat
FAILED_RETRYABLE definite provider failure and policy permits retry approved retry only
FAILED_MANUAL definite failure requiring correction/review owning operator action
UNKNOWN provider may have accepted provider reconciliation before retry
PROVIDER_ACCEPTED_LOCAL_PENDING provider object exists, local reference missing link/reconcile only
COMPENSATION_REQUIRED committed provider/physical effect needs reversal approved compensation workflow

5. Universal Unknown-Result Procedure

Step 1 — Freeze
Do not press Send, Sync, Start, Complete or Retry again.

Step 2 — Local business reread
Open the authoritative business record and confirm current state/version.

Step 3 — Effect-state reread
Read provider-specific status, request key/token, attempt and reference fields.

Step 4 — Provider evidence
Use the provider's supported lookup, callback history or current provider object.

Step 5 — Classify
Existing exactly once / definitely absent / multiple or ambiguous / unavailable.

Step 6 — Act
Link existing result, retry only after proven absence, or move to manual review.

Step 7 — Verify
Reread local and provider state and record evidence references.

If provider lookup is unavailable, keep UNKNOWN and escalate. Do not convert unknown into failed merely to enable Retry.

6. QBO Invoice Recovery

6.1 First read

Collect:

  • AR transaction UUIDs and billing group;
  • QBO_SYNC_STATUS;
  • QBO_INVOICE_ID;
  • QBO_DOC_NUMBER;
  • QBO_SYNC_TOKEN;
  • QBO_SYNCED_AT;
  • safe QBO_ERROR/claim disposition;
  • customer and amount/source fingerprint;
  • action time and actor.

6.2 State handling

SYNCED with QBO reference

Do not sync again.
-> Verify the QBO invoice matches the intended customer/lines/amount.
-> If correction is needed, use the finance/QBO correction process.

READY

No provider attempt is currently represented by the local state. Confirm no recent unknown attempt or provider object exists before executing if the operator previously saw a timeout.

SYNCING

Do not assume provider processing.
-> Determine claim age and current active request.
-> Check whether a QBO invoice already exists using the accepted matching process.
-> If one exists, link/reconcile.
-> If provider absence is proven and policy permits, release/retry through Billing.
-> Otherwise manual review.

ERROR

ERROR can mean definite provider rejection, local persistence failure or an ambiguous outcome depending on the path. Read the exact evidence before retry.

6.3 Provider accepted / local missing

QBO invoice exists
+ local QBO_INVOICE_ID absent
-> Do not create another invoice
-> verify one unambiguous provider match
-> link existing invoice through an approved reconciliation command/process

6.4 Multiple QBO matches

  • stop automation;
  • preserve all provider IDs;
  • do not delete/recreate automatically;
  • Finance determines the canonical invoice and duplicate disposition;
  • link/correct through the finance owner.

6.5 Escalation

Include transaction UUIDs, customer, amount, Doc Number candidate, QBO Invoice IDs, local status, action time and request/correlation IDs. Exclude OAuth tokens and raw Authorization data.

7. QuickBooks Time Recovery

7.1 First read

  • T_STAFF_TIME_EVENT source event;
  • staff user and QUICKBOOKS_TIME_USER_ID;
  • event type/time/job/break context;
  • current provider active timesheet;
  • provider result/log disposition;
  • token/config availability without secret values.

7.2 Authority

Local Staff Time event
= business/workforce authority

QuickBooks Time
= downstream provider projection

Do not change/delete the local event solely to match a provider error.

7.3 Active external timesheet conflict

If the provider reports an active timesheet not owned by the CTC event/jobcode policy:

  • leave the local event unchanged;
  • do not forcibly close the external timesheet automatically;
  • assign manual_review_required;
  • verify with the workforce/payroll owner;
  • only use an approved provider-specific correction.

7.4 Unknown provider write

  • reread the provider user's current timesheet;
  • compare event time/jobcode/notes/provider ID;
  • if exactly one matching CTC result exists, reconcile/link;
  • if absent is proven, retry through the owning Staff Time workflow;
  • if ambiguous, manual review.

7.5 Token refresh failure

Token refresh is separate from timesheet mutation.

Refresh failed before provider write
-> correct OAuth/config
-> reread provider state
-> execute only if no write attempt occurred

Never expose refreshed token values in an incident or Wiki.

8. Gmail Outgoing Recovery

8.1 Validate intended send

Before any resend, reconstruct:

  • business source;
  • recipient/CC/BCC target;
  • subject/template version;
  • required attachments and file versions;
  • intended thread/reply behavior;
  • prior Gmail Message/Thread ID;
  • local marker/reply ID/status.

8.2 Missing attachment

Current sender can skip a missing path. Therefore:

Send reported success
+ required attachment missing
= business email incomplete

Do not resend automatically until the process owner decides whether a corrected follow-up, replacement or full resend is appropriate.

8.3 Timeout or local marker failure

  • search the connected mailbox/sent thread using the strongest safe identity;
  • inspect stored Gmail message/thread IDs when available;
  • verify recipient, subject, time and attachment identity;
  • if exactly one sent message exists, persist/reconcile the local reference;
  • if provider absence is proven, deliberate resend may be allowed;
  • if multiple/ambiguous, manual review.

8.4 Cron mail

For late/no-show/ASN/report jobs:

  • reread the business marker (NOTIFICATION_TIME, SECOND_NOTIFICATION_TIME, ASN_TIME, etc.);
  • inspect the job-specific email log and Gmail sent state;
  • do not rerun the entire cron mode to repair one uncertain child;
  • send/correct the one missing notification through an approved procedure;
  • confirm the marker only after provider evidence is accepted.

9. Gmail Appointment Watch And Intake Recovery

9.1 Status checks

Read:

  • connected mailbox;
  • queue label;
  • watch expiration;
  • LAST_HISTORY_ID;
  • LAST_SYNC_AT;
  • LAST_NOTIFICATION_AT;
  • watch status/error;
  • target Gmail message/provider ID;
  • T_MAIL_MESSAGE and T_MAIL_REQUEST_ITEM state.

9.2 Duplicate/stale notification

A duplicate or lower/equal history notification can be safely treated as no-op when current watch state confirms it was already covered.

9.3 Missing baseline/history gap

WATCH_STATE_MISSING or baseline missing
-> register/re-establish watch through owner

Gmail history 404/gap
-> manual replay from accepted point or bounded recent-message replay
-> verify persisted message/item identities
-> advance cursor only after successful ingestion

9.4 Message ingested but no request item

  • inspect extraction/batch status and error;
  • verify whether the message legitimately produces no actionable item;
  • rerun only the intake/extraction owner for that message if approved;
  • do not resend or alter the Gmail message as a substitute.

9.5 Approved item but reply/mutation partial

Treat these independently:

Request item decision
Appointment mutation
Gmail reply send
Reply ID persistence

Reread each checkpoint and resume only the missing step.

10. Telegram Recovery

10.1 Interpret result

Result Meaning
ok=true, no suppression, provider message result provider reported success
ok=true, suppressed=true intentionally not sent
ok=false, definite provider error failed
timeout/network/invalid response potentially unknown; current sender may not fully distinguish

10.2 Loader/assignment notification

  • reread the assignment/planner row;
  • confirm whether Telegram-specific marker/result exists for that workflow;
  • verify the Loader page shows the assignment;
  • if Telegram was suppressed/failed, use the approved manual notification fallback;
  • do not recreate the assignment to resend a Telegram alert;
  • do not equate Telegram with Assignment creation or Start eligibility.

10.3 Webhook/callback

  • validate tenant, chat and linked staff user;
  • verify whether the action is informational/redirection or a business command;
  • use current business row state to determine whether a callback was applied;
  • do not rely only on the webhook response;
  • repeated callback must not recreate work.

10.4 Break cron

  • verify scheduler invocation time and tenant;
  • reread open/expired Break queues and Staff Time events;
  • do not rerun the cron blindly if a prior invocation may have partially processed items;
  • process/reconcile the remaining due rows through the owner.

11. AI Voice / Twilio / Gemini Recovery

11.1 Outbound call start

Read:

  • Load/transload;
  • AI Voice Case/Call UUID;
  • REQUEST_KEY and attempt;
  • local call status;
  • Twilio Call SID;
  • provider unknown/failure state;
  • callback timestamps/status;
  • Office follow-up Operation Event if present.

Existing Call with PROVIDER_UNKNOWN

Do not start another call for the Load/request key.
-> query/reconcile Twilio using Call SID or provider evidence where available
-> wait for callback within the accepted window
-> move to Office follow-up if still unresolved

Definite failure under attempt limit

Retry only through AI Voice owner and current request-key contract. After the current limit/manual threshold, use Office follow-up.

11.2 Callback failure

  • verify Twilio signature and target call UUID;
  • reread local Call state and provider SID;
  • inspect whether Twilio will retry after the returned status;
  • replay/application must be conditional and idempotent;
  • do not manually set final Call status without provider evidence and owner approval.

11.3 Voice gateway

Read:

  • service health/release/config state;
  • Twilio Stream SID and Call UUID;
  • gateway activity/session state;
  • nonce/replay state;
  • Office transfer state;
  • transcript sequence.

A failed gateway tool request does not authorize repeating complete_inbound_checkin without rereading the matched Load and Call session state.

11.4 Transcript outbox

Check:

  • gateway host/instance;
  • configured outbox directory;
  • file count and oldest age;
  • call UUID/sequence;
  • CTC transcript rows;
  • gateway delivery errors;
  • expiry.

Rules:

  • only one accepted drainer/ownership model per directory;
  • do not copy files between instances without preserving uniqueness and sequence;
  • if a transcript row already exists, remove/reconcile the matching file only through the outbox owner;
  • expired transcript evidence requires owner/privacy policy review; do not silently extend retention.

11.5 Recording cleanup

DELETE_FAILED is a provider cleanup result, not a failed call. Retry only the recording deletion. Do not delete the local Call/Transcript until the retention workflow's prerequisites are satisfied.

12. File Object / Metadata / Link Recovery

12.1 Required reread

  • T_FILE row and soft-delete state;
  • storage type and safe path/object key;
  • business link (TABLE_NAME, TABLE_ID, TYPE);
  • object existence/readability;
  • OBJECT_CLEANUP_AT and cleanup error/attempt evidence where available;
  • other active links to the same object;
  • archive location/manifest where applicable.

12.2 Object exists, metadata absent

This can result from object-first upload plus DB failure.

  • do not create arbitrary metadata from filename alone;
  • reconstruct tenant/source/type/UUID from the original command/evidence;
  • either complete registration through the owning file command or delete the true orphan under approved cleanup;
  • verify no other link/reference exists.

12.3 Metadata exists, object missing/unreadable

  • mark/support-classify UNREADABLE or DIVERGED;
  • do not claim upload success;
  • recover from accepted source/backup or request re-upload;
  • preserve business record and evidence of the missing object;
  • do not remove metadata solely to hide the inconsistency.

One object may be intended for multiple transactions/records.

  • read all intended and actual links;
  • add only missing links through the owner;
  • do not upload the same object again;
  • preserve one object identity where policy permits.

12.5 Cleanup pending

  • confirm metadata/link was intentionally soft-deleted;
  • confirm no remaining active links;
  • retry only physical object cleanup;
  • on success set cleanup marker/readback;
  • on repeated failure retain the pending row and escalate storage-specific diagnostics.

13. Worker / Scheduler Recovery

13.1 Process not running

  • verify deployment/service identity and release;
  • verify whether the worker is intentionally disabled;
  • inspect current ready/leased/in-flight/manual work from the provider-specific source;
  • start/restart only after understanding in-flight leases/attempts;
  • do not reset every item to READY.

13.2 Lease expired

No provider attempt started
-> safe re-claim if owner policy allows

Provider attempt may have started
-> UNKNOWN
-> reconcile before new attempt

13.3 Overlapping cron runs

Where no durable lease/run authority exists:

  • use source-record conditional predicates and timestamps to determine actual work;
  • stop overlapping scheduler triggers;
  • do not infer a complete run from one process exit;
  • reconcile each child family;
  • document scheduler/deployment evidence.

13.4 Partial batch

  • use per-item results;
  • never replay successful/suppressed/unknown items as a group;
  • retry only definite failed items allowed by policy;
  • move ambiguous items to manual review.

14. Webhook / Callback Recovery

Invalid signature/token

  • reject without applying business state;
  • do not expose verification details;
  • check provider configuration/secret rotation separately;
  • never weaken verification to restore traffic.

Duplicate callback

  • return the provider's accepted duplicate response only after confirming duplicate identity/current state;
  • no second business transition;
  • preserve one callback/evidence reference.

Out-of-order callback

  • compare provider event version/time and current local state;
  • do not regress a newer terminal state;
  • record ignored-stale disposition when supported.

Callback accepted but apply failed

If a durable receipt exists, retry application from that receipt. If no durable receipt exists, rely on provider retry/current state and provider-specific reconciliation; do not claim the callback is safely queued.

15. Manual Review Queue Procedure

For each unresolved item record:

provider/effect
business source
local state
effect/attempt/callback state
provider object/reference
time/age
reason for manual review
safe next action
owner/role
deadline/SLA if approved
evidence references

Resolution dispositions:

LINK_EXISTING_PROVIDER_RESULT
CONFIRM_ABSENCE_AND_RETRY
CORRECT_LOCAL_BUSINESS_STATE
CORRECT_PROVIDER_STATE
APPROVED_COMPENSATION
CANCEL_NOT_STARTED_INTENT
KEEP_ON_HOLD

Do not use a generic “Resolved” value without recording the actual disposition.

16. Escalation Matrix

Issue Primary owner Secondary owner
QBO duplicate/unknown invoice Billing/Finance Integration Support
QuickBooks Time conflict Workforce/Payroll Integration Support
Gmail outgoing unknown/attachment owning business process Gmail credential owner
Gmail watch/history gap Appointment/Mail owner Gmail credential/deployment owner
Telegram suppressed/failure Planning/Operations owner Telegram config owner
AI Voice provider unknown AI Voice/Office Twilio/Gateway owner
Transcript outbox backlog Voice Gateway owner Deployment/Storage owner
File object divergence File/Storage owner Business record owner
Callback signature/replay Provider integration owner Security/Deployment owner
Worker/cron overlap or outage Deployment/Operations Provider/business owner

17. Observation Checklist

For an approved provider/worker deployment observe:

  • oldest pending/unknown/manual-review age;
  • new intents and attempts by provider/tenant;
  • provider acceptance/reject/timeout/rate-limit;
  • local result persistence failures;
  • callback lag/duplicates/out-of-order;
  • worker claims/lease expiry/restarts;
  • retries and retry limit;
  • provider duplication incidents;
  • suppressed effects;
  • file object/metadata/link divergence;
  • transcript outbox backlog/expiry;
  • operator recovery time;
  • Client 1/2 non-regression;
  • source release and rollback state.

Observation does not authorize automatic retries.

18. Current Versus Target Tool Warning

Current tools differ by provider. The target consolidated support timeline/manual-review UI is not implemented.

Do not tell operators to use:

  • a universal Retry button;
  • a universal Outbox page;
  • a universal Provider Attempt table;
  • a universal Dead Letter queue;
  • a generic Worker dashboard;

until the relevant provider-scoped implementation is approved, deployed and verified.

19. Closure Criteria

Close an integration incident only when:

  • current local business state is known;
  • provider effect state is known or explicitly remains manual/unknown;
  • required provider/local references are linked;
  • no blind replay remains scheduled;
  • duplicate/partial results are addressed;
  • callback/worker/file residuals are understood;
  • next owner is explicit for any remaining work;
  • evidence is retained without secrets;
  • user/operator impact is communicated;
  • recurrence action is assigned to the correct provider/process owner.

Do not close solely because an alert disappeared or the provider eventually returned 2xx.

---

Source: docs/runbooks/PLATFORM_INTEGRATION_PROVIDER_RECOVERY_RUNBOOK.md — ctc repository · This page is an English translation of the source document. Identifiers, file paths and code symbols are unchanged.