Platform Integration, Outbox, Worker And Provider Reconciliation — Rebuild Packet¶

Status: DETAILED REBUILD INPUT / DOCUMENTATION ONLY
Current source baseline: qa/master@a5280b7cdc1c78b809ed6f1996ca31491b522146
Source authority: PLATFORM_INTEGRATION_OUTBOX_WORKER_PROVIDER_SOURCE_AUTHORITY.md
Audit/correlation authority: PLATFORM_AUDIT_TRANSITION_CORRELATION_SUPPORT_REBUILD_PACKET.md
Current policy: generic DB outbox/replay remains canceled unless a provider-scoped decision supersedes it
Target Runtime: current PHP adapters during coexistence, framework-independent TypeScript integration applications, separate Node.js worker only for explicitly approved provider/long-running families, MySQL first, PostgreSQL later
Phase implementation: USER HOLD
Runtime, DB schema, queue, worker deployment, provider call, callback, credential, Wiki publication and Production effects: NOT AUTHORIZED

1. Capability Outcome¶

The platform must let a business process request an external or long-running effect without confusing the effect with the business transaction.

For every approved effect family the system must be able to answer:

Why is the effect required?
Which local business record/process owns the request?
Which tenant, actor and subject authorized it?
What exact payload version/fingerprint was intended?
Was a provider attempt started?
Did the provider definitely reject, definitely accept or return an unknown outcome?
Was the provider result persisted locally?
Did a webhook/callback arrive, and was it applied exactly once?
Can the effect be retried automatically, manually or not at all?
What is the next safe owner action?
Which evidence proves the current boundary?

The target must support React + Next.js migration while preserving provider-specific truth and the current manual-recovery policy unless a narrower approved decision changes it.

1.1 Non-goals¶

This packet does not authorize:

one universal T_OUTBOX table with generic JSON payloads;
a universal automatic retry/replay button;
a universal dead-letter status replacing provider-specific records;
automatic replay of QBO invoice, Gmail, Telegram or physical/device effects;
moving business state transitions into a worker;
holding a database transaction open during a provider call;
storing credentials, raw provider payloads or message bodies in a generic attempt ledger;
treating a callback acknowledgement as business completion;
changing the accepted P01 manual-handling policy;
changing current provider clients or callers;
deploying a Node worker or queue;
introducing a new table or broker while Phase implementation is held.

2. Concern Model¶

flowchart LR
    CMD[Business Command]
    DB[(Business State)]
    INT[Effect Intent]
    ATT[Provider Attempt]
    PROV[Provider]
    CB[Callback Receipt]
    REC[Reconciliation]
    AUD[Transition Audit]
    SUP[Support Timeline]

    CMD --> DB
    CMD -. when approved .-> INT
    INT --> ATT
    ATT --> PROV
    PROV --> CB
    PROV --> REC
    CB --> REC
    REC --> DB
    CMD --> AUD
    ATT --> AUD
    REC --> AUD
    DB --> SUP
    INT --> SUP
    ATT --> SUP
    CB --> SUP
    AUD --> SUP

Concern	Target owner	Must not own
Business command/state	application/domain owner	provider retry scheduling
Effect intent	provider/effect application owner	generic arbitrary payload execution
Attempt	provider adapter/application	business approval or state transition meaning
Callback receipt	provider callback adapter	business state without owning application validation
Reconciliation	provider/effect application	unrelated business corrections
Worker	infrastructure runtime	business policy or tenant authorization definition
Notification	operation-event owner	provider delivery truth or approval
Audit/evidence	audit/domain owners	source data replacement
Support timeline	read model	writes, retry or compensation

3. Provider-Scoped Versus Shared Infrastructure¶

3.1 Provider-scoped authority¶

Each effect family defines its own:

effect type
owner process/application
business source identity
payload schema/version
provider idempotency identity
attempt/receipt fields
success/failure/unknown semantics
retry eligibility
manual recovery
retention/redaction
callback rules

Examples:

QBO invoice intent is owned by AR Billing;
QuickBooks Time intent is owned by Staff Time;
appointment reply is owned by the Mail/Appointment process;
Loader Telegram intent is owned by Planning/Assignment;
AI Voice call is owned by AI Voice Call/Case;
transcript persistence is owned by AI Voice transcript;
file object cleanup is owned by File/Archive cleanup.

3.2 Shared runtime plumbing¶

A future worker runtime may share:

process supervision and graceful shutdown;
tenant binding;
handler registry;
lease/heartbeat utility;
rate-limit utility;
request/process/audit correlation;
structured logging and metrics;
credential-owner resolution;
redaction;
bounded batch execution;
health/readiness endpoints.

It must not create a second authority for provider status or business completion.

3.3 Generic outbox prohibition¶

The following schema is not an accepted target:

T_OUTBOX
  TYPE
  PAYLOAD_JSON
  STATUS
  RETRY_COUNT

without a provider-scoped decision, typed schema and exact owner.

A typed shared physical table could be considered later only if all participating effect owners accept:

strict discriminator and schema versions;
tenant-first indexes;
no raw secret/large payload storage;
provider-specific unique/idempotency constraints;
separate typed result tables or fields;
explicit retention;
no cross-provider replay semantics;
provider-specific Runbook and rollback.

This packet does not approve that design.

4. Common Typed Contracts¶

These are logical contracts. They do not require one common database table.

4.1 `EffectIntent`¶

interface EffectIntent {
  schemaVersion: string;
  effectIntentId: string;
  tenant: TenantReference;
  ownerProcess: ProcessReference;
  ownerCommand: CommandReference;
  effectType: string;
  businessSource: EntityReference;
  commandRequestToken: string;
  correlationId: string;
  auditId?: string;
  payloadVersion: string;
  payloadFingerprint: string;
  credentialOwnerRef: CredentialOwnerReference;
  requiredBy: string;
  createdAt: string;
  state: EffectIntentState;
}

Rules:

payloadFingerprint covers a canonical safe payload, not credentials;
identical command token + effect type + source + payload version/fingerprint returns the same intent;
same identity with a different fingerprint is a conflict;
tenant and credential owner must agree;
intent creation does not mean provider delivery;
intent may be in the same business transaction only when the selected family has an approved durable model.

4.2 `EffectAttempt`¶

interface EffectAttempt {
  providerAttemptId: string;
  effectIntentId: string;
  provider: string;
  attemptNumber: number;
  workerRunId?: string;
  leaseToken?: string;
  startedAt: string;
  completedAt?: string;
  requestFingerprint: string;
  providerRequestRef?: string;
  providerResultRef?: string;
  outcome: ProviderAttemptOutcome;
  failureCode?: string;
  safeFailureRef?: string;
  nextEligibleAt?: string;
}

An attempt is immutable except for a bounded transition from in-flight to one terminal/unknown outcome. Do not overwrite prior attempts with the latest error.

4.3 `CallbackReceipt`¶

interface CallbackReceipt {
  callbackReceiptId: string;
  provider: string;
  tenant: TenantReference;
  providerEventId?: string;
  providerObjectRef?: string;
  signatureState: 'VALID' | 'INVALID' | 'UNAVAILABLE';
  receivedAt: string;
  payloadFingerprint: string;
  duplicateOf?: string;
  applyState: 'RECEIVED' | 'APPLIED' | 'IGNORED' | 'FAILED' | 'UNKNOWN';
  failureCode?: string;
}

Callback receipt storage, if approved, stores bounded scalar metadata and a safe fingerprint/reference. Raw provider bodies remain in provider-specific controlled storage/logging only when required.

4.4 `ReconciliationResult`¶

interface ReconciliationResult {
  effectIntentId: string;
  disposition:
    | 'PROVIDER_FOUND_AND_LINKED'
    | 'PROVIDER_ABSENT_RETRY_ALLOWED'
    | 'PROVIDER_REJECTED'
    | 'MULTIPLE_MATCHES_MANUAL_REVIEW'
    | 'LOCAL_RESULT_MISSING'
    | 'CALLBACK_PENDING'
    | 'UNKNOWN_MANUAL_REVIEW';
  providerObjectRef?: string;
  evidenceReferences: EvidenceReference[];
  nextAction: string;
  reconciledAt: string;
  reconciledBy: ActorReference;
}

5. State Models¶

5.1 Effect intent state¶

NOT_REQUIRED
READY
CLAIMED
IN_FLIGHT
PROVIDER_ACCEPTED_LOCAL_PENDING
SUCCEEDED
FAILED_RETRYABLE
FAILED_MANUAL
UNKNOWN
COMPENSATION_REQUIRED
CANCELED
SUPPRESSED

Definitions:

State	Meaning	Allowed next owner action
`NOT_REQUIRED`	policy says no effect is needed	none
`READY`	approved local intent exists	claim if family allows execution
`CLAIMED`	one worker/request owns the execution lease	execute or release/expire
`IN_FLIGHT`	provider request started	await result/callback; do not replay
`PROVIDER_ACCEPTED_LOCAL_PENDING`	provider acceptance known, local persistence incomplete	reconcile/link only
`SUCCEEDED`	provider result and required local reference persisted	no repeat
`FAILED_RETRYABLE`	definite failure and policy permits retry	retry at/after policy time
`FAILED_MANUAL`	definite failure requiring operator action	manual review
`UNKNOWN`	provider may have accepted	reconcile before retry
`COMPENSATION_REQUIRED`	external/physical effect requires approved compensation	no ordinary retry
`CANCELED`	intent was canceled before unsafe effect	no execution
`SUPPRESSED`	effect intentionally not sent by config/policy	show separately from success

5.2 Provider attempt outcome¶

NOT_STARTED
STARTED
ACCEPTED
REJECTED
TIMEOUT_UNKNOWN
CONNECTION_UNKNOWN
INVALID_RESPONSE_UNKNOWN
LOCAL_PERSIST_FAILED
CALLBACK_PENDING
SUCCEEDED

5.3 Worker job state¶

A provider-scoped durable job, when approved, uses:

AVAILABLE
LEASED
RUNNING
WAITING_RETRY
WAITING_CALLBACK
MANUAL_REVIEW
COMPLETED
CANCELED
EXPIRED

A worker job state never replaces the effect state. It describes execution mechanics only.

5.4 Callback state¶

RECEIVED
DUPLICATE
OUT_OF_ORDER
APPLIED
IGNORED
FAILED_RETRYABLE
FAILED_MANUAL
UNKNOWN

6. Stable Identity And Idempotency¶

6.1 Common identity chain¶

requestId
-> commandRequestToken
-> processInstanceId / correlationId
-> auditId
-> effectIntentId
-> providerAttemptId
-> provider object/event/message/call/file reference
-> callbackReceiptId

6.2 `effectKey`¶

An effect family defines a stable canonical key such as:

QBO_INVOICE:<tenant>:<billing-group>:<invoice-version>
GMAIL_BOL:<tenant>:<transload>:<document-version>:<recipient-set>
PACKING_EMAIL:<tenant>:<booking>:<target>:<part>:<document-version>
TELEGRAM_ASSIGNMENT:<tenant>:<planner>:<loader>
QBTIME_EVENT:<tenant>:<staff-event>
AI_VOICE_CALL:<tenant>:<call-request-key>
AI_TRANSCRIPT:<tenant>:<call-uuid>:<sequence>
FILE_OBJECT_CLEANUP:<tenant>:<storage>:<object-key>

The exact key belongs to the provider/effect family. A generic worker may not invent or change it.

6.3 Provider idempotency¶

Classify each provider operation:

PROVIDER_IDEMPOTENT
PROVIDER_QUERY_RECONCILABLE
LOCAL_IDEMPOTENT_ONLY
NON_IDEMPOTENT / MANUAL_REVIEW_REQUIRED
UNKNOWN

Retry policy cannot be selected until this classification is accepted.

6.4 Callback idempotency¶

Preferred identity order:

provider event ID;
provider object ID + event type + provider timestamp/version;
canonical safe payload fingerprint;
provider-specific nonce/request identity.

A callback with no reliable identity may be processed only through an owning state transition predicate that makes duplicates harmless, or it must remain manual/unsupported.

7. Transaction Boundaries¶

7.1 Local transaction with durable intent¶

Allowed only for a provider family with an approved durable model.

begin business transaction
-> validate command and state
-> update business row
-> insert typed effect intent
-> append transition audit when required
-> commit

Provider calls happen after commit.

7.2 Direct synchronous compatibility¶

Current direct callers may remain during migration.

local claim or business commit
-> provider call in request
-> local result save

The adapter must classify:

definite reject;
accepted and saved;
accepted/local-save-pending;
unknown result;
suppressed.

7.3 Prohibited transaction¶

begin DB transaction
-> call provider for seconds/minutes
-> wait for callback
-> commit DB

Long provider calls must not hold business locks or leases.

7.4 File object boundary¶

object write
!= metadata/link transaction

The selected family defines either:

object-first with compensation;
intent-first with worker upload;
direct signed upload with finalization command.

Each needs explicit orphan/unknown recovery.

8. Worker Runtime Contract¶

8.1 Runtime role¶

A future separate Node.js worker may run:

provider delivery handlers;
callback follow-up handlers;
provider reconciliation;
document/file long-running generation;
bounded cleanup/retention;
cron-like due work;
support package generation.

It does not run ordinary business queries merely because they are asynchronous.

8.2 Worker identity¶

Every worker execution requires:

deploymentId
workerType
workerInstanceId
release SHA
config release ID where applicable
tenant binding or selected tenant
handler version
workerRunId
request/correlation ID

8.3 Lease contract¶

A durable job family, when approved, defines:

leaseOwner
leaseToken
leasedAt
leaseExpiresAt
heartbeatAt
attemptNumber
nextEligibleAt

Rules:

conditional lease acquisition;
lease ownership verified before result persistence;
provider call occurs without a long DB transaction;
lost lease does not imply provider failure;
expired lease with an in-flight/unknown provider attempt goes to reconciliation, not direct replay;
one tenant/job cannot starve the global worker;
graceful shutdown stops new claims and finishes or safely releases current work.

8.4 Concurrency¶

Worker concurrency is bounded by:

provider rate limits;
tenant limits;
effect-family ordering;
source-record serialization requirements;
callback ordering;
local DB capacity;
file/device resource limits.

8.5 Batch behavior¶

Batch results are per item.

SUCCEEDED
SKIPPED
SUPPRESSED
FAILED_RETRYABLE
FAILED_MANUAL
UNKNOWN

One child failure must not be hidden by a global success=true. All-or-nothing behavior requires an explicit business policy and cannot compensate an already committed provider effect automatically.

8.6 Health/readiness¶

A worker exposes:

/healthz
= process alive

/readiness
= dependencies required for new work are available

Readiness reports safe capability states, not credentials.

8.7 Worker observability¶

Required dimensions:

provider/effect family;
tenant/deployment;
handler/release version;
queue/ready/manual/unknown counts;
claim latency;
execution duration;
retry count;
callback delay;
provider rate-limit/reject/unknown;
oldest pending/manual-review age;
reconciliation result;
graceful shutdown outcome.

Metrics do not become provider or business authority.

9. Retry And Backoff Policy¶

9.1 Retry classification¶

Condition	Retry disposition
local validation/authorization failure	no automatic retry
provider definite 4xx business rejection	manual correction; no same-payload retry
provider definite rate limit with safe idempotency	bounded delayed retry
connection timeout with unknown provider outcome	reconcile first
local result save failure after known provider acceptance	link/reconcile only
worker process crash before provider request	retry if no attempt started
lease expired after provider request start	reconcile first
callback processing transient DB failure	provider-specific callback retry or durable receipt retry
callback invalid signature	reject; never retry internally
file cleanup transient failure	bounded provider-specific retry
physical effect unknown	manual/owner reconciliation; no auto replay

9.2 Backoff¶

Provider-specific retry defines:

initial delay
exponential/linear rule
maximum delay
maximum automatic attempts
provider Retry-After handling
jitter
business deadline
manual-review threshold

No global retry count is accepted for every provider.

9.3 Dead letter/manual review¶

Prefer the term MANUAL_REVIEW for business-visible unresolved effects.

A technical dead-letter store may exist in a future worker, but:

it cannot be the only source of truth;
it references the provider-specific intent/attempt;
moving to dead letter does not cancel business work;
replay requires provider-specific authorization and reconciliation;
secrets/raw payloads remain excluded.

10. Webhook And Callback Contract¶

10.1 Security¶

Every callback specifies:

provider signature/token/HMAC verification;
accepted timestamp skew;
replay/nonce policy;
expected tenant/provider binding;
allowed method/content type/body size;
source IP policy only when provider supports reliable ranges;
secret rotation and overlap policy;
safe error response.

10.2 Fast acknowledgement¶

Two accepted models:

Apply synchronously¶

Use only when processing is bounded, idempotent and completes within provider timeout.

verify
-> conditional apply
-> commit
-> acknowledge

Durable receipt then async apply¶

Use when callback processing is long-running or provider retry windows are strict.

verify
-> persist bounded callback receipt
-> acknowledge
-> worker applies provider-specific transition

This model requires a provider-scoped callback receipt decision and is not automatically enabled.

10.3 Duplicate/out-of-order handling¶

Callback application must compare:

provider object identity;
provider event identity/version/time;
current local provider state;
allowed state transition;
prior callback receipts/attempts.

An older event does not overwrite a newer terminal state unless the provider contract explicitly requires it.

10.4 Callback result¶

APPLIED
DUPLICATE
IGNORED_STALE
IGNORED_UNRELATED
FAILED_RETRYABLE
FAILED_MANUAL

Do not return provider success while silently failing a required local transition unless a durable receipt guarantees later processing.

11. Provider Family Target Contracts¶

11.1 QBO invoice¶

Target provider-scoped capabilities:

CreateQboInvoiceIntent
ClaimQboInvoiceAttempt
DeliverQboInvoice
ReconcileQboInvoice
LinkExistingQboInvoice
MarkQboInvoiceManualReview

Required before automatic retry:

accepted invoice identity/matching key;
provider query support;
stale SYNCING policy;
provider duplicate search;
exact finance role/approval;
immutable attempt references;
local/provider reconciliation.

Until then, manual finance recovery remains authoritative.

11.2 QuickBooks Time¶

Target capabilities:

ProjectStaffTimeEvent
ReconcileActiveTimesheet
ApplyQuickBooksTimeEffect
ResolveQuickBooksTimeConflict

The local Staff Time event remains authoritative. External active-timesheet conflict stays manual review unless an approved policy defines a safe merge/close.

Token refresh and timesheet mutation are separate effect stages.

11.3 Gmail outgoing¶

Target capabilities:

PrepareOutgoingMail
ValidateOutgoingAttachments
SendOutgoingMail
ReconcileOutgoingMail
ResumeMissingRecipientPart

One outgoing mail effect identity includes:

business source;
recipient target/set;
template/body version or fingerprint;
attachment object/version set;
thread/reply intent;
request token.

Missing attachment is an error when required; the provider adapter must not silently omit it.

11.4 Gmail appointment intake¶

Preserve current provider-specific authority:

Mailbox Watch State
Mail Message
Request Item
History Cursor
Manual Replay

A future worker may execute sync/replay, but it must use the same durable cursor/work items and not introduce another mailbox queue.

11.5 Telegram¶

Target capabilities:

SendTelegramNotification
EditTelegramMessage
HandleTelegramCallback
ReconcileTelegramResult

Effect result includes:

SENT
SUPPRESSED
REJECTED
UNKNOWN

Before durable retry, define provider message identity, duplicate-message impact and current business criticality. Generic replay remains disallowed.

Incoming Update ID/callback identity requires a provider-scoped replay decision if duplicate callbacks become harmful.

11.6 AI Voice / Twilio / Gemini¶

Preserve current strong patterns:

local Call/Case request identity before provider call;
known versus unknown provider result;
callback-driven state;
bounded nonce replay protection;
provider recording cleanup state;
transcript sequence idempotency;
Office fallback.

Target improvements may include:

profile/capability selection instead of numeric Client ID;
durable callback receipts where callback complexity requires it;
shared/multi-instance transcript outbox ownership;
explicit gateway deployment/release identity;
typed provider attempts referenced by transition audit;
provider-specific support timeline.

Do not replace the call/case domain model with a generic outbox.

11.7 Files¶

Target capabilities:

CreateFileObjectIntent
FinalizeFileMetadataAndLink
VerifyFileReadability
CleanupFileObject
ReconcileFileObjectState

Required result distinguishes:

OBJECT_STORED
METADATA_STORED
LINKS_COMPLETE
READABLE
CLEANUP_PENDING
CLEANED
UNKNOWN

A file worker, if approved, owns object transport/cleanup only. The business owner authorizes the link and retention.

12. Query Contracts¶

GetIntegrationCapabilityStatus
GetProviderCredentialAvailability
GetEffectIntent
GetEffectAttemptHistory
GetProviderObjectReconciliation
GetCallbackReceiptHistory
GetWorkerQueueSummary
GetWorkerRunDetail
GetManualReviewQueue
GetIntegrationSupportTimeline
GetFileObjectReconciliation
GetMailboxWatchState
GetVoiceGatewayOutboxStatus

Every query is tenant-scoped and returns safe references rather than credentials or raw provider payloads.

12.1 `GetEffectIntent`¶

Returns:

intent identity/state;
owner process/command/source;
payload version/fingerprint;
provider/effect type;
current attempt summary;
callback/reconciliation summary;
exact next safe action;
evidence references;
current release/deployment/handler version;
retention/redaction status.

12.2 `GetManualReviewQueue`¶

Filters by:

tenant/provider/effect;
age/severity;
owner process/role;
unknown/manual/compensation state;
provider object/reference availability.

It does not expose other tenants or raw secret/provider content.

13. Command Contracts¶

CreateProviderEffectIntent
CancelProviderEffectIntent
ClaimProviderEffectAttempt
RecordProviderAttemptResult
RecordProviderCallbackReceipt
ApplyProviderCallback
ReconcileProviderEffect
ApproveProviderRetry
ApproveProviderCompensation
MoveProviderEffectToManualReview
ResolveProviderManualReview
RetryProviderEffect
PurgeProviderEffectEvidence

These are semantic target commands, not one universal API.

13.1 `RetryProviderEffect`¶

Preconditions:

selected provider/effect policy allows retry;
prior result is definitely failed or provider absence is confirmed;
current business source is still eligible;
payload/version/fingerprint is unchanged or a new intent is created;
actor has provider-specific retry capability;
no active lease/attempt/callback is pending;
retry limit/deadline is not exceeded.

It must return the same intent with a new immutable attempt, not mutate history.

13.2 `ApproveProviderCompensation`¶

Requires:

effect boundary is externally/physically committed or compensation-required;
explicit owner/supervisor approval;
compensation command belongs to the provider/business owner;
original and compensation effects are linked;
no destructive deletion of original evidence.

14. Stable Error Catalog¶

INTEGRATION_UNAUTHENTICATED
INTEGRATION_FORBIDDEN
INTEGRATION_PROFILE_NOT_APPLICABLE
INTEGRATION_MODULE_DISABLED
INTEGRATION_CAPABILITY_DISABLED
INTEGRATION_CREDENTIAL_UNAVAILABLE
INTEGRATION_CREDENTIAL_OWNER_MISMATCH
INTEGRATION_CONFIG_INVALID

EFFECT_INTENT_NOT_FOUND
EFFECT_INTENT_CONFLICT
EFFECT_DUPLICATE_REQUEST
EFFECT_PAYLOAD_CONFLICT
EFFECT_CANCELED
EFFECT_SUPPRESSED
EFFECT_ALREADY_COMPLETE
EFFECT_IN_FLIGHT
EFFECT_CLAIM_CONFLICT
EFFECT_LEASE_LOST
EFFECT_RETRY_NOT_ALLOWED
EFFECT_RETRY_LIMIT_EXCEEDED
EFFECT_RECONCILIATION_REQUIRED
EFFECT_MANUAL_REVIEW_REQUIRED
EFFECT_COMPENSATION_REQUIRED

PROVIDER_REJECTED
PROVIDER_RATE_LIMITED
PROVIDER_TIMEOUT_UNKNOWN
PROVIDER_CONNECTION_UNKNOWN
PROVIDER_INVALID_RESPONSE
PROVIDER_ACCEPTED_LOCAL_PENDING
PROVIDER_OBJECT_NOT_FOUND
PROVIDER_MULTIPLE_MATCHES
PROVIDER_RESULT_CONFLICT

CALLBACK_INVALID_SIGNATURE
CALLBACK_REPLAY_REJECTED
CALLBACK_DUPLICATE
CALLBACK_OUT_OF_ORDER
CALLBACK_TENANT_MISMATCH
CALLBACK_APPLY_FAILED
CALLBACK_RESULT_UNKNOWN

WORKER_UNAVAILABLE
WORKER_HANDLER_NOT_FOUND
WORKER_RELEASE_MISMATCH
WORKER_QUEUE_UNAVAILABLE
WORKER_LEASE_CONFLICT
WORKER_SHUTDOWN_IN_PROGRESS
WORKER_RESULT_PERSIST_FAILED

FILE_OBJECT_STORE_FAILED
FILE_METADATA_STORE_FAILED
FILE_LINK_PARTIAL
FILE_OBJECT_UNREADABLE
FILE_OBJECT_CLEANUP_FAILED
FILE_OBJECT_LOCAL_STATE_DIVERGED

WEBHOOK_BODY_INVALID
WEBHOOK_PROVIDER_UNAVAILABLE
INTEGRATION_DATA_CONFLICT
INTEGRATION_UNEXPECTED_ERROR

User-visible messages remain bounded. Internal credentials, stack traces, provider payloads and SQL do not appear in the public error.

15. UI And Operator Experience¶

15.1 React/Current UI states¶

Every provider effect UI must distinguish:

Not Required
Ready / Pending
In Progress
Waiting for Callback
Succeeded
Suppressed
Definitely Failed
Retry Available
Unknown — Do Not Retry
Manual Review
Compensation Required
Canceled

15.2 Required visible fields¶

business source and provider/effect type;
current local business state;
effect state;
last safe provider reference;
attempt count and last attempt time;
whether provider outcome is known;
callback/reconciliation state;
next safe action and owner;
current release/handler where relevant;
no-blind-retry warning;
evidence links allowed to the role.

15.3 Prohibited UI¶

one green Success that hides partial child effects;
Retry button for unknown outcomes;
Delivered label for Telegram suppression;
Synced inferred only from HTTP 200;
raw provider error/secret display;
generic “Fix” button that mutates multiple owners;
cross-tenant provider search.

16. Operations And Recovery¶

The detailed procedure is in PLATFORM_INTEGRATION_PROVIDER_RECOVERY_RUNBOOK.md.

Universal safe sequence:

stop replay
-> identify tenant/source/effect/request
-> reread local business state
-> reread effect intent/attempt state
-> inspect provider/callback evidence
-> classify definite fail / success / unknown / partial
-> resume only the missing owner step
-> record manual/compensation decision

No provider action is repeated solely because the browser, cron or worker response was missing.

17. Migration And Coexistence¶

17.1 Waves¶

INT-W0  Current provider/caller/credential/scheduler/deployment characterization
INT-W1  Typed read model and support diagnostics
INT-W2  Correlation and safe evidence-reference parity
INT-W3  Provider-specific intent/attempt contract for one selected family
INT-W4  Provider-specific worker read shadow
INT-W5  One execution cohort one-writer cutover
INT-W6  Callback receipt/apply cutover where required
INT-W7  Unknown-result reconciliation and manual-review UI
INT-W8  Target-role/provider stabilization and observation
INT-W9  Current direct caller-zero for selected family
INT-W10 Additional provider families only through separate approval
INT-W11 Shared/Dedicated worker deployment parity
INT-W12 PostgreSQL later

17.2 First candidate selection¶

Do not select the first provider solely because it is easy to call.

Selection criteria:

evidence of operational pain or migration dependency;
current durable source identity;
provider idempotency/query support;
manageable effect volume;
safe manual fallback;
target-role availability;
no physical safety risk;
one bounded caller/cohort;
explicit Product/operations approval.

AI Voice transcript already has a provider-specific outbox and may be a useful infrastructure characterization case, but it is not automatically the first business migration.

17.3 One writer¶

For a selected effect/cohort:

Current PHP caller
or
Target worker/application
= one authoritative execution path

Read shadow and reconciliation shadow are allowed. Dual provider execution is prohibited.

17.4 Rollback¶

Rollback:

stops new target claims;
preserves all intents/attempts/provider references/callbacks;
drains or freezes in-flight work according to provider policy;
reconciles unknown provider outcomes;
restores routing to the prior caller only for not-started/safe effects;
keeps target evidence for support;
never deletes provider objects/messages/invoices/calls merely to restore code routing;
never replays already accepted effects.

17.5 Caller-zero¶

Caller-zero requires:

static search for old provider client/caller paths;
runtime observation over an accepted window;
zero old writer executions for the selected cohort;
old cron/webhook route disposition;
current credential and config ownership migrated;
rollback window closed or superseded;
Runbook/Wiki updated;
provider-specific evidence preserved.

18. PostgreSQL Preservation¶

A later PostgreSQL implementation must preserve:

tenant-first identity and indexes;
UUID/request/effect/attempt uniqueness;
canonical payload fingerprint semantics;
conditional lease/state transitions;
immutable attempt history;
callback deduplication identity;
provider object references;
timestamp precision and UTC normalization;
JSON only for bounded typed provider metadata where approved;
no raw credentials/payloads in generic tables;
foreign-key or application constraints without weakening provider-specific recovery;
archive/purge/legal-hold policy.

Do not combine the provider worker cutover with the first PostgreSQL cutover.

19. Stable Scenario Catalog¶

19.1 Common effect scenarios¶

INT-SCN-010  Local business commit; no effect required
INT-SCN-020  Intent created; provider not called
INT-SCN-030  Definite provider success and local save
INT-SCN-040  Definite provider rejection
INT-SCN-050  Provider timeout with unknown result
INT-SCN-060  Provider accepted; local save failed
INT-SCN-070  Duplicate command token / same payload
INT-SCN-080  Duplicate command token / different payload
INT-SCN-090  Concurrent claims
INT-SCN-100  Lease expires before provider call
INT-SCN-110  Lease expires after provider call starts
INT-SCN-120  Rate-limited safe retry
INT-SCN-130  Retry limit -> manual review
INT-SCN-140  Suppressed effect
INT-SCN-150  Compensation required

19.2 Callback scenarios¶

INT-CB-SCN-010 valid callback applied
INT-CB-SCN-020 invalid signature rejected
INT-CB-SCN-030 duplicate callback
INT-CB-SCN-040 out-of-order callback
INT-CB-SCN-050 callback for unknown object
INT-CB-SCN-060 callback DB transient failure
INT-CB-SCN-070 provider retries after 500
INT-CB-SCN-080 tenant/provider mismatch
INT-CB-SCN-090 callback received before local provider result save

19.3 Provider scenarios¶

QBO-SCN-P-010 invoice accepted/local save complete
QBO-SCN-P-020 invoice accepted/local save unknown
QBO-SCN-P-030 QBO search finds one existing invoice
QBO-SCN-P-040 multiple QBO matches -> manual review
QBTIME-SCN-010 local event/provider success
QBTIME-SCN-020 external active timesheet conflict
GMAIL-SCN-010 outgoing mail and required attachment success
GMAIL-SCN-020 provider accepted/local marker missing
GMAIL-SCN-030 Pub/Sub duplicate notification
GMAIL-SCN-040 history gap -> manual replay
TELEGRAM-SCN-010 message sent
TELEGRAM-SCN-020 delivery suppressed
TELEGRAM-SCN-030 timeout/unknown result
VOICE-SCN-010 local call intent/Twilio accepted
VOICE-SCN-020 provider unknown/no retry
VOICE-SCN-030 callback duplicate/out-of-order
VOICE-SCN-040 transcript outbox restart/drain
FILE-SCN-P-010 object/metadata/link/readability complete
FILE-SCN-P-020 object stored/metadata failed
FILE-SCN-P-030 metadata soft-deleted/object cleanup failed

19.4 Migration scenarios¶

INT-MIG-SCN-010 read/support parity
INT-MIG-SCN-020 provider-specific intent shadow
INT-MIG-SCN-030 one-writer cutover
INT-MIG-SCN-040 in-flight work during cutover
INT-MIG-SCN-050 rollback before provider call
INT-MIG-SCN-060 rollback after provider acceptance
INT-MIG-SCN-070 callback routing coexistence
INT-MIG-SCN-080 old cron/caller disabled
INT-MIG-SCN-090 worker restart/lease recovery
INT-MIG-SCN-100 Shared/Dedicated deployment parity
INT-MIG-SCN-110 PHP caller-zero
INT-MIG-SCN-120 PostgreSQL parity later

20. Evidence Template¶

For each scenario record:

source SHA and deployment/release
provider sandbox/production mode
Tenant/profile/module/capability
actor/subject/role
business source ID/UUID
command request token
process/correlation/audit ID
effect intent ID
provider attempt ID
provider object/event/message/call/file reference
callback receipt ID
worker run/instance/lease identity
before/after local state
provider response classification
exact readback
retry/reconciliation/compensation action
cleanup/residual state
logs/evidence references with redaction
result and reviewer

Never store the credential or raw sensitive payload in the evidence package.

21. Open Policy Decisions¶

Shared runtime¶

whether a shared Node worker is required before a provider-specific trigger exists;
process manager/container/service model;
Shared/Dedicated worker topology;
queue/storage technology if any;
lease/heartbeat defaults;
tenant fairness;
graceful shutdown and deploy drain.

Provider-specific durability¶

first approved provider/effect family;
exact intent/attempt persistence owner;
provider idempotency/search identity;
callback receipt requirement;
retry/backoff/limit;
manual-review owner;
compensation policy;
retention and purge.

Current families¶

QBO invoice matching/search and stale SYNCING policy;
QuickBooks Time conflict and token-refresh ownership;
Gmail outgoing attempt/result persistence;
required attachment enforcement;
Gmail watch replay ownership and scheduler;
Telegram incoming Update ID deduplication and suppressed UI;
AI Voice numeric tenant restriction retirement;
voice transcript outbox multi-instance ownership;
callback receipt persistence for Twilio;
file cleanup worker ownership and orphan policy;
cron run/lease/observation authority.

No implementation begins by answering these questions implicitly in code.

22. Definition Of Ready¶

A provider/effect implementation wave is ready only when:

business owner/effect owner are explicit;
current caller and source state are source-reconciled;
effect key and request identity are accepted;
provider idempotency/query/reconciliation semantics are known;
definite failure versus unknown outcome is defined;
retry/manual/compensation owner is approved;
credential owner and tenant binding are explicit;
callback/security/deduplication is defined where applicable;
one-writer and rollback are accepted;
UI/Runbook/Wiki wording is prepared;
target-role/provider/deployment fixtures are available;
no generic outbox policy is being revived implicitly.

23. Definition Of Complete¶

A selected provider/effect wave is complete only when:

target local and provider states are correct;
duplicate/timeout/unknown/callback/worker-restart scenarios pass;
exact local/provider readback is available;
support timeline can explain the current boundary;
manual review and compensation are usable;
secrets are redacted;
worker deployment/health/metrics are observed;
rollback is rehearsed without provider replay;
old writer/caller reaches zero;
current Wiki/Runbook is published and role-accepted;
stabilization and value are observed;
evidence is retained according to accepted policy.

This documentation packet alone does not satisfy these conditions.

---¶

Source: docs/reconstruction/PLATFORM_INTEGRATION_OUTBOX_WORKER_PROVIDER_REBUILD_PACKET.md — ctc repository · This page is an English translation of the source document. Identifiers, file paths and code symbols are unchanged.

Platform Integration, Outbox, Worker And Provider Reconciliation — Rebuild Packet¶

1. Capability Outcome¶

1.1 Non-goals¶

2. Concern Model¶

3. Provider-Scoped Versus Shared Infrastructure¶

3.1 Provider-scoped authority¶

3.2 Shared runtime plumbing¶

3.3 Generic outbox prohibition¶

4. Common Typed Contracts¶

4.1 EffectIntent¶

4.2 EffectAttempt¶

4.3 CallbackReceipt¶

4.4 ReconciliationResult¶

5. State Models¶

5.1 Effect intent state¶

5.2 Provider attempt outcome¶

5.3 Worker job state¶

5.4 Callback state¶

6. Stable Identity And Idempotency¶

6.1 Common identity chain¶

6.2 effectKey¶

6.3 Provider idempotency¶

6.4 Callback idempotency¶

7. Transaction Boundaries¶

7.1 Local transaction with durable intent¶

7.2 Direct synchronous compatibility¶

7.3 Prohibited transaction¶

7.4 File object boundary¶

8. Worker Runtime Contract¶

8.1 Runtime role¶

8.2 Worker identity¶

8.3 Lease contract¶

8.4 Concurrency¶

8.5 Batch behavior¶

8.6 Health/readiness¶

8.7 Worker observability¶

9. Retry And Backoff Policy¶

9.1 Retry classification¶

9.2 Backoff¶

9.3 Dead letter/manual review¶

10. Webhook And Callback Contract¶

10.1 Security¶

10.2 Fast acknowledgement¶

Apply synchronously¶

Durable receipt then async apply¶

10.3 Duplicate/out-of-order handling¶

10.4 Callback result¶

11. Provider Family Target Contracts¶

11.1 QBO invoice¶

11.2 QuickBooks Time¶

11.3 Gmail outgoing¶

11.4 Gmail appointment intake¶

11.5 Telegram¶

11.6 AI Voice / Twilio / Gemini¶

11.7 Files¶

12. Query Contracts¶

12.1 GetEffectIntent¶

12.2 GetManualReviewQueue¶

13. Command Contracts¶

13.1 RetryProviderEffect¶

13.2 ApproveProviderCompensation¶

14. Stable Error Catalog¶

15. UI And Operator Experience¶

15.1 React/Current UI states¶

15.2 Required visible fields¶

15.3 Prohibited UI¶

16. Operations And Recovery¶

17. Migration And Coexistence¶

17.1 Waves¶

17.2 First candidate selection¶

17.3 One writer¶

17.4 Rollback¶

17.5 Caller-zero¶

18. PostgreSQL Preservation¶

19. Stable Scenario Catalog¶

19.1 Common effect scenarios¶

19.2 Callback scenarios¶

19.3 Provider scenarios¶

19.4 Migration scenarios¶

20. Evidence Template¶

4.1 `EffectIntent`¶

4.2 `EffectAttempt`¶

4.3 `CallbackReceipt`¶

4.4 `ReconciliationResult`¶

6.2 `effectKey`¶

12.1 `GetEffectIntent`¶

12.2 `GetManualReviewQueue`¶

13.1 `RetryProviderEffect`¶

13.2 `ApproveProviderCompensation`¶