This is an internal working record, not an everyday instruction. It was written while the system was being rebuilt, for whoever maintains it. If you are here to do today's work, use the everyday documents instead: Procedures · Processes · Runbooks · The working day, step by step

Platform Integration, Outbox, Worker And Provider Reconciliation — Policy And Sequence

• Status: CURRENT POLICY RECONCILIATION / TARGET DESIGN INPUT / DOCUMENTATION ONLY
• Source baseline: qa/master@a5280b7cdc1c78b809ed6f1996ca31491b522146
• Source authority: PLATFORM_INTEGRATION_OUTBOX_WORKER_PROVIDER_SOURCE_AUTHORITY.md
• Rebuild packet: PLATFORM_INTEGRATION_OUTBOX_WORKER_PROVIDER_REBUILD_PACKET.md
• Existing accepted policy: P01 External Effect Manual Handling Policy
• Phase implementation: USER HOLD
• Runtime, schema, provider, callback, worker, Wiki and deployment effects: NONE

---

1. Stable Policy Decisions

INT-POL-010 — Generic outbox remains canceled

Status: ACCEPTED CURRENT POLICY

CTC does not currently adopt a generic DB outbox/replay queue for all Gmail, Telegram, QBO or QuickBooks Time effects.

A future provider-scoped durable intent/attempt model requires a new decision identifying:

• provider/effect;
• business owner;
• current source of truth;
• duplicate/unknown risk;
• provider idempotency/query support;
• manual fallback;
• retention/access;
• schema/deployment authorization.

INT-POL-020 — Worker is infrastructure, not business authority

Status: TARGET DIRECTION

A separate Node.js worker may execute approved work. The application/domain owner remains responsible for:

• whether an effect is required;
• payload meaning;
• business source and state eligibility;
• retry/compensation policy;
• local reconciliation;
• completion and user result.

INT-POL-030 — Provider call outside long business transaction

Status: TARGET REQUIREMENT

Do not hold a business DB transaction open across a network call or callback wait.

INT-POL-040 — Unknown outcome requires reconciliation

Status: ACCEPTED CROSS-PROCESS DIRECTION

Timeout / lost response / worker crash after request start
-> UNKNOWN
-> provider and local reread
-> retry only when provider absence is confirmed or idempotency is accepted

INT-POL-050 — Suppressed is not delivered

Status: CURRENT SEMANTIC CORRECTION

Telegram and any future configurable effect must expose:

SUPPRESSED

separately from SUCCEEDED/SENT.

INT-POL-060 — Callback acknowledgement is not callback application

Status: TARGET REQUIREMENT

A provider-facing 2xx can mean only that the callback was safely accepted or durably recorded. It must not be used as evidence that the business transition completed unless the local apply transaction actually committed.

INT-POL-070 — Provider attempts are immutable history

Status: TARGET REQUIREMENT

Do not overwrite the previous attempt with the latest error. Each approved provider attempt has its own identity and outcome.

INT-POL-080 — Business source remains authoritative

Status: ACCEPTED DIRECTION

Examples:

• Staff Time event remains authority over QuickBooks Time;
• AR Billing remains authority over QBO invoice request;
• Appointment request item remains authority over Gmail intake approval;
• AI Voice Call remains authority over Twilio call lifecycle;
• File owner remains authority over business link and retention.

INT-POL-090 — Provider-specific manual review is a first-class result

Status: TARGET REQUIREMENT

Use MANUAL_REVIEW when safe automation cannot determine the result. Do not conceal it as failed, queued or completed.

INT-POL-100 — No automatic provider replay from a generic support screen

Status: ACCEPTED SAFETY RULE

A support screen may navigate to the owning provider workflow. It does not execute a generic replay/compensation command.

INT-POL-110 — Callback and webhook security is provider-specific

Status: TARGET REQUIREMENT

Twilio signature, Gmail Pub/Sub token, Telegram secret and CTC voice-gateway HMAC/nonce are different contracts. Do not replace them with one weak shared token convention.

INT-POL-120 — File object, metadata, link and readability remain separate

Status: ACCEPTED DIRECTION

A worker may move bytes or clean objects, but only the business/file owner creates or removes links and decides retention.

INT-POL-130 — Voice transcript filesystem outbox is provider-specific

Status: CURRENT SOURCE FACT

The voice gateway outbox is a valid bounded mechanism for transcript delivery. It is not an approved template for QBO, Gmail, Telegram or arbitrary business work.

INT-POL-140 — Current cron scripts are schedulable commands, not durable queues

Status: CURRENT SOURCE FACT

External scheduler invocation, HTTP 200 or CLI exit does not prove every eligible child was processed. Job-run/lease history is not assumed where it is absent.

INT-POL-150 — Shared/Dedicated deployment must preserve tenant and credential ownership

Status: TARGET REQUIREMENT

A worker or callback must identify:

• deployment;
• tenant;
• credential owner;
• handler/release version;
• effect family.

A deployment fallback credential cannot silently cross tenant boundaries.

---

2. Policy Decisions Still Open

ID	Decision	Why it matters	Owner
INT-POL-210	first provider/effect selected for durable worker	prevents speculative generic platform build	Product/Operations/P6/P12
INT-POL-220	shared worker service topology	Shared/Dedicated and availability	P9/P12
INT-POL-230	queue/lease persistence technology	durability, recovery and PostgreSQL portability	P5/P6/P12/P11
INT-POL-240	QBO provider reconciliation identity	prevents duplicate invoice	Billing/Finance
INT-POL-250	QBO stale SYNCING timeout and owner	defines safe recovery	Billing/Finance
INT-POL-260	outgoing Gmail durable result requirements	message/attachment/recipient correctness	Process owner/P8
INT-POL-270	Telegram retry/deduplication necessity	avoids duplicate alerts and false success	Planning/Operations
INT-POL-280	Gmail watch scheduler/replay owner	mailbox continuity	Appointment/Operations
INT-POL-290	Twilio callback receipt persistence	callback durability and ordering	AI Voice owner
INT-POL-300	voice transcript outbox multi-instance model	host-local files can split ownership	AI Voice/P9
INT-POL-310	File cleanup worker and orphan policy	object/data consistency	Files/Archive
INT-POL-320	cron run/lease authority	overlapping runs and missed execution	Operations/P9
INT-POL-330	manual-review role and SLA by provider	unresolved effects need accountable owner	Product/Operations
INT-POL-340	provider evidence retention/legal hold	cleanup must not destroy required evidence	Data/Legal/Product

No implementation may silently answer an open policy through a default constant or generic status.

---

3. Current Direct Provider Sequence

sequenceDiagram
    actor User
    participant UI as Current PHP/JS UI
    participant App as Application Helper
    participant DB as MySQL
    participant Provider

    User->>UI: Submit action
    UI->>App: HTTP request
    App->>DB: Validate / claim / commit local state
    App->>Provider: Direct synchronous request
    alt definite provider success
        Provider-->>App: provider object/result
        App->>DB: Save provider reference/result
        App-->>UI: Success
    else definite provider failure
        Provider-->>App: rejection/error
        App->>DB: Save error when owner supports it
        App-->>UI: Failure/manual action
    else timeout or lost response
        Provider--xApp: unknown outcome
        App->>DB: ERROR/SYNCING/UNKNOWN depending on family
        App-->>UI: Unknown or generic failure
    end

Critical boundary:

Provider may have committed before the response or local result save.

---

4. Current QBO Invoice Sequence

sequenceDiagram
    actor BillingUser
    participant Billing as AR Billing
    participant DB as MySQL
    participant QBO as QuickBooks Online

    BillingUser->>Billing: Sync invoice group
    Billing->>DB: Conditional READY/ERROR -> SYNCING claim
    DB-->>Billing: claim token / group
    Billing->>QBO: Create invoice
    alt provider success and local save success
        QBO-->>Billing: Invoice ID / Doc Number / Sync Token
        Billing->>DB: Save QBO fields, set SYNCED
        Billing-->>BillingUser: Synced
    else provider rejects
        QBO-->>Billing: Definite error
        Billing->>DB: Set ERROR
        Billing-->>BillingUser: Error
    else provider accepted but response/local save fails
        QBO--xBilling: Unknown/partial
        Billing->>DB: ERROR or stale SYNCING may remain
        Billing-->>BillingUser: Ambiguous result
        Note over BillingUser,QBO: Do not create again before provider reconciliation
    end

---

5. Current Gmail Outgoing Sequence

sequenceDiagram
    actor UserOrCron
    participant Owner as Business Caller
    participant Gmail as GmailSender/API
    participant DB as MySQL
    participant Log as JSONL

    UserOrCron->>Owner: Send email action/job
    Owner->>Owner: Build recipient/body/attachments
    Owner->>Gmail: users.messages.send
    alt reported success
        Gmail-->>Owner: Message ID / Thread ID
        Owner->>DB: Save marker/reply ID when caller supports it
        Owner->>Log: Write result
    else reported failure
        Gmail-->>Owner: Error result
        Owner->>Log: Write error
    else provider accepted / local marker failed
        Gmail-->>Owner: Provider success
        Owner--xDB: Marker persistence fails
        Note over Owner,DB: Local pending can cause later duplicate send
    end

Required attachment validation occurs before GmailSender; the sender currently skips missing paths.

---

6. Gmail Pub/Sub Intake Sequence

sequenceDiagram
    participant Gmail as Gmail/PubSub
    participant Hook as Watch Service
    participant State as T_MAILBOX_WATCH_STATE
    participant Intake as Intake Service
    participant Mail as T_MAIL_MESSAGE / REQUEST_ITEM

    Gmail->>Hook: Push mailbox + history ID
    Hook->>Hook: Validate token and mailbox
    Hook->>State: Read last history ID
    alt duplicate/stale
        Hook-->>Gmail: No-op success
    else missing baseline or history gap
        Hook->>State: Mark ERROR / needs replay
        Hook-->>Gmail: Replay-required result
    else newer history
        Hook->>Gmail: List history pages
        loop changed messages
            Hook->>Intake: Ingest message
            Intake->>Mail: Transactional message + immutable item batch
        end
        Hook->>State: Advance history and mark ACTIVE
        Hook-->>Gmail: Synced
    end

---

7. Current Telegram Outbound Sequence

sequenceDiagram
    participant Owner as Planning/Break/Other Owner
    participant Sender as TelegramSender
    participant Telegram

    Owner->>Sender: Send/Edit/Callback action
    alt tenant/module/config disabled
        Sender-->>Owner: ok=true, suppressed=true
        Note over Owner: Must display SUPPRESSED, not SENT
    else enabled
        Sender->>Telegram: Bot API request (short timeout)
        alt provider response
            Telegram-->>Sender: ok/result
            Sender-->>Owner: provider result
        else network/invalid response
            Telegram--xSender: timeout/error
            Sender-->>Owner: ok=false / unknown details
        end
    end

No common durable attempt is created by the sender.

---

8. Current Telegram Webhook Sequence

sequenceDiagram
    participant Telegram
    participant Hook as Telegram Webhook
    participant DB as MySQL
    participant Sender as TelegramSender

    Telegram->>Hook: Message or callback + shared secret
    Hook->>Hook: Verify secret and parse action
    alt ignored action
        Hook-->>Telegram: ok / ignored
    else relevant action
        Hook->>DB: Resolve exactly one linked staff user
        Hook->>Sender: Answer/edit/send Loader-page message
        Sender-->>Hook: provider result
        Hook-->>Telegram: handled result
    end

Current source does not establish a durable Telegram Update ID receipt ledger.

---

9. Current AI Voice Start And Callback Sequence

sequenceDiagram
    actor Office
    participant API as AI Voice API
    participant DB as MySQL
    participant Twilio
    participant Callback as Twilio Status Callback

    Office->>API: Start call with REQUEST_KEY
    API->>DB: Create/reuse local Call intent in transaction
    DB-->>API: Call UUID / attempt / state
    alt existing intent
        API-->>Office: Existing call state
    else new intent
        API->>Twilio: Create Call
        alt definite success
            Twilio-->>API: Call SID/status
            API->>DB: Save provider result
            API-->>Office: Queued
        else definite failure
            Twilio-->>API: Rejection
            API->>DB: Mark failure
            API-->>Office: Retry or manual follow-up
        else unknown
            Twilio--xAPI: Timeout/ambiguous response
            API->>DB: Mark PROVIDER_UNKNOWN
            API-->>Office: Do not retry; confirm status
        end
    end
    Twilio->>Callback: Status/recording callback
    Callback->>Callback: Validate provider signature
    Callback->>DB: Conditional provider-state transaction
    Callback-->>Twilio: TwiML/HTTP result

---

10. Current Voice Transcript Filesystem Outbox

sequenceDiagram
    participant Gateway as Node Voice Gateway
    participant FS as Secure Outbox Directory
    participant CTC as Signed CTC Transcript Endpoint
    participant DB as Transcript Segment Store

    Gateway->>FS: Stage call/sequence/speaker/text JSON atomically
    Gateway->>CTC: Deliver (up to 3 immediate attempts)
    alt accepted
        CTC->>DB: Idempotent segment append
        CTC-->>Gateway: Stored
        Gateway->>FS: Delete entry
    else unavailable
        Gateway--xCTC: Failure/timeout
        Note over FS: Entry retained
    end
    loop every 30 seconds
        Gateway->>FS: Scan bounded batch
        Gateway->>CTC: Redeliver unexpired entries
        alt accepted
            Gateway->>FS: Delete
        else expired/invalid
            Gateway->>FS: Delete and log
        end
    end

Deployment concern:

Host-local directory
!= multi-instance shared queue

---

11. Current File Object / Metadata / Cleanup Sequence

sequenceDiagram
    actor User
    participant FileAPI
    participant Storage as EFS/S3
    participant DB as T_FILE / Business Link

    User->>FileAPI: Upload
    FileAPI->>Storage: Store object
    alt object stored
        FileAPI->>DB: Register metadata/link
        alt DB success
            FileAPI-->>User: File result
        else DB failure
            FileAPI->>Storage: Attempt compensation delete
            FileAPI-->>User: Failure / possible orphan object
        end
    else object failure
        FileAPI-->>User: Storage failure
    end

    User->>FileAPI: Delete link/object
    FileAPI->>DB: Soft-delete metadata/link
    FileAPI->>Storage: Delete object when policy permits
    alt cleanup success
        FileAPI->>DB: Set OBJECT_CLEANUP_AT
    else cleanup failure
        FileAPI-->>User: Local delete may be complete; cleanup remains pending
    end

---

12. Target Provider-Scoped Worker Sequence

sequenceDiagram
    actor User
    participant App as Application Command
    participant DB as Business + Provider-specific Intent
    participant Worker
    participant Provider
    participant Audit

    User->>App: Execute business command
    App->>DB: Validate and commit business row + typed intent
    App->>Audit: Append transition/effect reference when required
    App-->>User: Local result + provider PENDING
    Worker->>DB: Conditional claim / lease
    Worker->>Provider: Provider-specific request
    alt definite success
        Provider-->>Worker: Provider object/result
        Worker->>DB: Save immutable attempt + provider ref + SUCCEEDED
        Worker->>Audit: Append effect evidence when policy requires
    else definite retryable failure
        Provider-->>Worker: Definite error/rate limit
        Worker->>DB: FAILED_RETRYABLE + nextEligibleAt
    else unknown
        Provider--xWorker: Timeout/lost response
        Worker->>DB: UNKNOWN / reconciliation required
    end

No long DB transaction spans the provider call.

---

13. Target Callback Receipt Sequence

sequenceDiagram
    participant Provider
    participant Hook as Provider Callback Adapter
    participant Receipt as Provider-specific Receipt Store
    participant Worker as Callback Apply Worker
    participant App as Owning Application
    participant DB as Business/Provider State

    Provider->>Hook: Signed callback
    Hook->>Hook: Verify signature/tenant/body/identity
    Hook->>Receipt: Conditional receipt insert
    alt duplicate
        Receipt-->>Hook: Existing receipt
        Hook-->>Provider: 2xx duplicate accepted
    else stored
        Hook-->>Provider: 2xx receipt accepted
        Worker->>Receipt: Claim unapplied receipt
        Worker->>App: Apply provider transition
        App->>DB: Conditional state/result transaction
        App->>Receipt: Mark APPLIED or manual/retry
    end

This sequence is target-only and requires provider-scoped approval.

---

14. Unknown Result Decision Flow

flowchart TD
    A[Provider action response missing or local result save failed] --> B[Stop replay]
    B --> C[Read local business state]
    C --> D[Read intent and attempt]
    D --> E[Query provider or inspect callback/provider evidence]
    E --> F{Provider object found?}
    F -- Exactly one --> G[Link existing object / persist result]
    F -- None, absence proven --> H{Retry policy allows?}
    H -- Yes --> I[Approved new attempt]
    H -- No --> J[Manual review]
    F -- Multiple or ambiguous --> J
    F -- Query unavailable --> J
    G --> K[Exact local/provider readback]
    I --> K
    J --> L[Escalation / compensation owner]

---

15. Worker Restart And Lease Flow

sequenceDiagram
    participant W1 as Worker A
    participant DB as Job/Attempt Store
    participant Provider
    participant W2 as Worker B

    W1->>DB: Claim job with lease
    W1->>Provider: Start request
    W1--xDB: Crashes before result persistence
    Note over DB: Lease eventually expires; attempt may be UNKNOWN
    W2->>DB: Inspect expired lease and attempt state
    alt provider request never started
        W2->>DB: Safe new claim
    else provider may have received request
        W2->>DB: Mark/retain reconciliation required
        W2->>Provider: Provider-specific lookup only
        W2->>DB: Link result or manual review
    end

---

16. Manual Review And Compensation Flow

flowchart LR
    U[UNKNOWN / FAILED_MANUAL / COMPENSATION_REQUIRED] --> R[Authorized provider-specific review]
    R --> E[Gather local/provider/evidence references]
    E --> D{Disposition}
    D -->|Existing provider result| L[Link/reconcile]
    D -->|Definite absence + retry approved| T[New immutable attempt]
    D -->|Business correction required| C[Owning business command]
    D -->|External effect must be reversed| A[Approved compensation command]
    D -->|Still ambiguous| H[Keep manual hold / escalate]

---

17. Cutover Policy

INT-W0 — Characterization

• exact callers and scheduled invocations;
• credentials/config/deployment;
• provider sandbox/production mode;
• current effect volume and manual recovery;
• current callbacks and rate limits;
• current residual/in-flight state.

INT-W1 — Safe read model

Build read-only diagnostics from existing provider-specific state. Do not write new intents or attempts.

INT-W2 — Provider-scoped intent contract

Only after INT-POL-210 selects a family. Define exact effect key, typed payload, state, attempt and reconciliation.

INT-W3 — Worker read shadow

Worker evaluates eligible work and proposed action but does not call the provider.

INT-W4 — One-writer canary

One tenant/effect/cohort routes provider execution through the target worker/application.

INT-W5 — Callback/reconciliation cutover

Only if required by the provider family.

INT-W6 — Stabilization

Target-role, provider, restart, timeout, rate-limit, callback and rollback scenarios.

INT-W7 — Caller-zero

Retire old direct callers/cron/webhook routes for the selected family after observation.

---

18. Rollback Policy

Rollback does not delete provider objects or replay business commands.

stop new target claims
-> inventory READY/LEASED/IN_FLIGHT/UNKNOWN/MANUAL work
-> preserve attempt and callback evidence
-> reconcile provider-accepted or unknown items
-> route only not-started/safe work to old caller
-> retain target support read model
-> verify no dual execution

A callback endpoint cannot be switched blindly if the provider may still deliver to the old URL. Provider configuration, overlap, receipt identity and deduplication must be planned.

---

19. Stable Scenario IDs

Use the scenario families defined in the Rebuild Packet:

INT-SCN-*
INT-CB-SCN-*
QBO-SCN-P-*
QBTIME-SCN-*
GMAIL-SCN-*
TELEGRAM-SCN-*
VOICE-SCN-*
FILE-SCN-P-*
INT-MIG-SCN-*

Each result records local state, provider evidence, effect/attempt identity, callback result, next safe action and residual state.

---

20. Stop Conditions

Stop documentation or implementation escalation when:

• a generic outbox is being assumed despite the accepted canceled policy;
• provider idempotency/reconciliation semantics are unknown;
• effect ownership is not tied to one business process/application;
• a retry could duplicate an invoice/message/call/physical effect;
• credential owner or tenant binding is ambiguous;
• callback verification or deduplication is undefined;
• worker topology would lose host-local filesystem outbox data;
• file object/link/cleanup authority is mixed;
• one-writer and rollback are not explicit;
• source/provider/deployment evidence is required to decide policy;
• a support UI would become a second mutation authority;
• Phase implementation is still held and code/schema/deployment work is proposed.

Record the exact provider/effect, unresolved decision, owner and evidence required.

---

Source: docs/processes/PLATFORM_INTEGRATION_OUTBOX_WORKER_PROVIDER_POLICY_AND_SEQUENCE.md — ctc repository · This page is an English translation of the source document. Identifiers, file paths and code symbols are unchanged.