Skip to content

Platform Tenant, Profile, Module, Capability And Configuration — Administrator And Support Runbook

  • Status: SOURCE-RECONCILED REPOSITORY RUNBOOK / CURRENT ADMIN AND TARGET RELEASE PATH SEPARATED
  • Current source baseline: qa@4f6165bc4c53a7d98b0c246392dd9ed5bd75c21c
  • Source authority: PLATFORM_TENANT_PROFILE_CAPABILITY_CONFIGURATION_SOURCE_AUTHORITY.md
  • Rebuild packet: PLATFORM_TENANT_PROFILE_CAPABILITY_CONFIGURATION_REBUILD_PACKET.md
  • Audience: super-admin, tenant administrator, configuration release owner, deployment operator and read-only support
  • Important: current Admin Console operations and target Control Tower release procedures are different. This document does not authorize a current module/config change or reveal secret values.

1. Purpose

Use this Runbook to determine why a tenant, page, API, workflow or provider behaves differently and to prepare a safe configuration change without confusing:

Tenant
Profile
Module
Capability
Config
Credential
Menu/Page/API
Action/Record

The first task is diagnosis and authoritative reread, not toggling settings.

2. Roles

2.1 Super Administrator

May inspect multiple tenants and use explicitly allowed cross-tenant admin routes.

Required:

  • name the target tenant;
  • preserve actor versus impersonated subject;
  • record authorization/correlation reference where required;
  • verify local/global mapping before portable operation;
  • avoid using tenant 0 as a business data context.

2.2 Tenant Administrator

May manage only the current tenant through the permitted Admin Console paths.

Must not:

  • select another tenant ID;
  • edit platform-global definitions;
  • infer provider readiness from a saved config field;
  • bypass a disabled module/capability.

Approved integration-credential management

An authenticated tenant administrator (role 1) may reveal and update only the allowlisted, tenant-owned integration credential fields on Provider Settings for the effective tenant. Super-administrators may do the same for an explicitly selected tenant. This permission does not apply to login passwords, browser sessions, user personal credentials, arbitrary environment variables, arbitrary config keys, or a disabled module/capability.

2.3 Configuration Release Owner

Owns target draft/review/publish/apply/rollback, not ordinary business operations.

2.4 Support Operator

Read-only diagnosis by default.

May collect safe metadata, screenshots and IDs. Must not change module/config state while diagnosing unless separately authorized.

2.5 Deployment Operator

Applies one approved release to one named deployment target and records exact local readback.

3. Before Any Diagnosis

Record:

date/time and timezone
current source/deployed SHA
deployment type: shared SaaS or dedicated
environment and database identity
target local tenant ID
target global tenant UUID/mapping state, if available
actor and subject
access mode: direct / impersonation / admin override / system
page/API/job/provider involved
current request/correlation ID
expected business result

Do not record:

  • passwords;
  • OAuth tokens;
  • API keys;
  • secret config values;
  • authorization headers;
  • full cookies/session data;
  • raw provider credential payloads.

4. Diagnose Effective Tenant

Step 4.1 — Identify entrypoint

WEB page
API route
CLI job
Integration callback

Step 4.2 — Identify tenant provenance

For WEB/API:

  1. check whether impersonation is active;
  2. identify original actor and target subject separately;
  3. identify session tenant;
  4. check whether the route uses an explicit allowlisted admin override;
  5. check whether the route opts into strict tenant principal.

For job/integration:

  1. identify explicit local tenant binding;
  2. identify global mapping if used;
  3. verify one tenant child context;
  4. reject reliance on deployment fallback for a new write.

Step 4.3 — Classify context

STRICT + MAPPED
STRICT + LEGACY_LOCAL_ONLY
COMPATIBILITY_ONLY
MISMATCH / INVALID

Stop

Stop if:

  • tenant target is ambiguous;
  • scalar and typed tenant differ;
  • global/local mapping conflicts;
  • impersonation actor/subject is malformed;
  • a consequential write would use compatibility-only context.

5. Diagnose Operating Profile

Current profile may not be a first-class Runtime assignment.

Step 5.1 — Separate fact from inference

Documented process/profile expectation:
ctc-execution or mk-order

Current Runtime evidence:
modules + configs + routes + record classes + numeric compatibility

Do not report an inferred profile as an explicit tenant assignment.

Step 5.2 — Record profile disposition

EXPLICIT ASSIGNMENT VERIFIED
COMPATIBILITY INFERENCE
MISSING
INVALID / VERSION CONFLICT
UNKNOWN

Step 5.3 — Check non-implications

Profile does not prove:

  • module enabled;
  • named workflow enabled;
  • role authorized;
  • config valid;
  • credential available.

6. Diagnose Module State

For the exact module key:

  1. verify module catalog row;
  2. verify tenant-module row;
  3. record IS_ENABLED, tier and lifecycle metadata without modifying it;
  4. identify current caller policy;
  5. identify page/API module mapping;
  6. identify source/DB read errors.

Classify:

EXPLICIT_ENABLED
EXPLICIT_DISABLED
UNCONFIGURED_COMPATIBILITY
SOURCE_ERROR_COMPATIBILITY
INVALID
UNKNOWN

Important current rule

A missing tenant-module row or module-source error may currently allow access. Do not change this globally during diagnosis.

Exact readback

Read:

  • module catalog key;
  • tenant assignment row or confirmed absence;
  • page/module mapping;
  • route module key;
  • observed current decision/reason;
  • target-role page/API result when separately authorized.

7. Diagnose Named Process Capability

Ask:

What exact behavior is being requested?
Which process/module owns it?
Is it narrower than the module?
Which profile applies?
Which config/credential prerequisites exist?

Examples:

render MK documents
create Export Brokerage job
apply inventory allocation
reverse Scale ticket
deliver QBO invoice

Current source may not expose a named capability result. In that case record:

CURRENT AD HOC POLICY

Do not fabricate an enabled/disabled capability from module state alone.

8. Diagnose Config Value

Step 8.1 — Identify the exact key and owner

Record:

  • config key;
  • owning module/provider/process;
  • expected type;
  • whether secret;
  • allowed fallback sources;
  • applicable profiles;
  • current caller.

Step 8.2 — Identify source and validity

Classify:

TENANT_CONFIG / VALID
PUBLISHED_SNAPSHOT / VALID
DEPLOYMENT_ENV / VALID FALLBACK
LEGACY_DEFAULT / COMPATIBILITY
CALLER_DEFAULT / COMPATIBILITY
MISSING
INVALID
SOURCE_ERROR
UNKNOWN

Step 8.3 — Do not infer from scalar result

A returned false, 0, empty string or default can mean several different current conditions. Inspect safe metadata/source behavior before concluding.

Step 8.4 — Numeric compatibility

When behavior depends on Client 1/2 or another numeric tenant branch, record:

compatibility selector
actual business difference
proposed profile/capability/config owner
caller removal condition

Do not add another numeric branch.

9. Diagnose Credential Availability

Never read or display the secret value during ordinary support diagnosis.

Verify:

credential key/reference
owner module
owner tenant local/global identity
source type
encrypted/external/env availability
module result
capability result
provider caller

Classify:

AVAILABLE / OWNER MATCHED
UNAVAILABLE / MISSING
UNAVAILABLE / INVALID
UNAVAILABLE / SOURCE ERROR
UNAVAILABLE / OWNER MISMATCH
DEPLOYMENT FALLBACK

A valid credential does not authorize a provider call.

9.1 Approved credential reveal/update procedure

This procedure is limited to a named provider credential that is already in the Provider Settings allowlist.

  1. Confirm the effective tenant and an authorized Administrator or Super-administrator session.
  2. Select the credential and use Show. The browser sends a POST request for that credential only; it is never part of the initial HTML or provider list.
  3. Check the value on screen, then use Hide before moving to another credential. Hide and page exit clear the displayed field value from the page state.
  4. To replace it, type the new non-empty value and select the page-level Save action. Empty input is no change; removal is not implicit.
  5. The update response never returns the supplied value.

Provider Settings changes use one POST batch. The server validates submitted modules, non-secret config keys, and allowlisted credentials, then applies them inside one database transaction. A validation or write failure rolls back the entire batch; the browser disables Provider Settings controls until it receives the result.

When provider modules are split or renamed, apply the matching tenant-module migration before deploying the source change. Gemini, Twilio, and Telegram runtime gates fail closed when their tenant module row is absent.

The reveal route is a POST-only, allowlisted credential endpoint under the tenant Admin API. It rechecks the existing Admin Console role/session and effective tenant server-side, returns one requested value only, and sends Cache-Control: no-store, private plus Pragma: no-cache. The existing API transport has no separate CSRF-token contract; it keeps the same-origin authenticated-session model used by the rest of Admin Console.

Tenant secrets are stored as SECRET.<CONFIG_KEY> values encrypted with the server-only CTC_TENANT_SECRET_KEY AES-256-GCM utility. Updating a legacy plaintext credential writes the encrypted companion row and wipes/retire the legacy row in the same transaction. Key rotation is performed by replacing the server-side key under the deployment secret-management process and re-encrypting credential rows through an approved maintenance operation; never expose the key or credential values in a runbook, ticket, console log, trace, analytics event, or audit payload.

Request/audit evidence may retain actor, tenant, allowlisted credential key, action (reveal/update), request ID, timestamp, IP, and success/failure. It must never retain plaintext, ciphertext, encryption key, request body, response body, exception text containing a value, or browser-console output. For an incident, rollback is to save the previously approved credential value through the same page; do not restore a raw value from logs.

10. Diagnose Menu / Page / API / Action / Record

Build one row for the failing/expected action.

Gate Evidence
Authentication session/system actor
Tenant strict/scalar context and mapping
Menu route visibility/role mapping
Page role/page guard and module page mapping
API method/path/strict principal/module/menu/role
Capability named behavior result or current policy
Config/Credential typed or current compatibility result
Record tenant ownership and state predicate
UI control visible/hidden/disabled
Result allow/deny/unknown and safe reason

Required parity:

menu visibility
== page load
== initial data API
== action control
== action API
== expected record result

Do not solve a mismatch by granting Administrator globally.

11. Current Admin Console — Read Procedure

Current Admin Console can inspect:

  • tenant identity;
  • module assignments/tier;
  • config rows;
  • provider definitions/settings;
  • onboarding controls.

Read sequence

  1. confirm deployment and target tenant;
  2. open Admin Console through the authorized route;
  3. select the exact tenant or current tenant-admin view;
  4. load module/config/provider tabs;
  5. record safe rows and counts, not secret values;
  6. compare to page/API/process behavior;
  7. classify explicit/compatibility/missing/error.

Dedicated deployment

The page presents tenant management as read-only. Treat this as current UI intent until API/server enforcement is independently verified.

12. Current Admin Console — Write Procedure

Use only with explicit authorization naming:

  • tenant;
  • exact module/config/identity field;
  • before state;
  • intended after state;
  • affected pages/APIs/processes;
  • rollback value;
  • observation owner.

12.1 Before write

  1. reread exact target row or absence;
  2. capture safe before metadata;
  3. verify no concurrent release/apply owner;
  4. check Client 1/2 impact;
  5. verify current route and role authority;
  6. identify cache/restart behavior;
  7. prepare rollback.

12.2 Execute once

Perform only the named module/config/tenant command.

Do not combine unrelated changes merely because they share the Admin page.

12.3 Required readback

API result
+ exact local row reread
+ module/config effective helper result
+ selected page/API behavior
+ cache/version/restart result

Current direct write does not prove Control Tower release publication.

12.4 Unknown response

If response is missing:

  1. reread the exact row;
  2. reread effective resolution;
  3. determine committed/not committed/unknown;
  4. repeat only when absence is proven;
  5. never resubmit an entire tenant settings batch blindly.

13. Target Release Procedure

This is a target procedure and is not current Runtime functionality unless separately implemented and accepted.

Draft
-> impact preview
-> reviewer approval
-> immutable publish
-> named deployment apply
-> local row readback
-> effective resolver readback
-> selected page/API/process check
-> observation
-> rollback/supersede when needed

Required release content:

  • tenant mapping;
  • profile version;
  • modules;
  • capabilities;
  • typed config;
  • credential references only;
  • expected applied version;
  • impact/rollback;
  • reviewer/approver;
  • source revision.

14. Common Problems And Safe Recovery

Problem: Page disappeared after module change

Read:

  • tenant module row;
  • page map and registry;
  • menu/role row;
  • API module decision;
  • current caller expectation.

Recovery:

  • restore exact prior module state only when authorized;
  • do not globally enable all modules;
  • record unknown callers.

Problem: Page opens but action API returns forbidden

Read:

  • page/menu role;
  • API required menu routes/role IDs;
  • module/capability;
  • record predicate.

Recovery:

  • correct the bounded mismatch;
  • do not make every user Administrator.

Problem: Config saved but behavior did not change

Read:

  • row and value type;
  • caller key/source/fallback;
  • static/process cache;
  • profile/capability prerequisites;
  • source SHA/deployment.

Recovery:

  • invalidate/restart only the accepted consumer;
  • do not rewrite the value repeatedly.

Problem: Behavior changed after missing config

Check whether a caller default or numeric tenant compatibility applies. Document the fallback source.

Problem: Provider says credential missing

Check safe availability/owner/module/capability; never copy secret values into logs or chat.

Problem: Super-admin sees different behavior while impersonating

Separate actor, subject, target tenant, client 0 bypass and impersonation status. Unimpersonated client-0 bypass must not be assumed under impersonation.

Problem: Dedicated tenant settings appear read-only

Verify deployment type and server/API enforcement before concluding that all writes are impossible.

15. Escalation Packet

Collect:

source/deployed SHA
deployment/environment
local/global tenant identity and mapping state
actor/subject/access mode
page/API/job/provider
menu/page/module/capability/config key
safe module/config/credential resolution
published/applied release IDs if any
before/after row metadata
request/correlation ID
exact error code/reason
selected readback result
cache/restart state
Client 1/2 impact
attempted recovery
rollback readiness

Exclude secret values and raw auth/session payloads.

16. Prohibited Actions

  • changing current missing-module behavior globally;
  • adding numeric Client/customer branches;
  • using Config::CLIENT_ID for a new write;
  • treating profile as authorization;
  • treating module as named capability;
  • treating config presence as provider permission;
  • copying secrets between tenants;
  • logging decrypted secrets;
  • changing several module/config rows during diagnosis;
  • retrying an unknown Admin write without reread;
  • bypassing record tenant/state predicates;
  • publishing Wiki instructions from target-only controls;
  • claiming Control Tower publication from a direct Runtime row update.

17. Current Evidence Boundary

Source and prior P4 evidence: AVAILABLE
Current DB/config values: NOT READ IN THIS WAVE
Current Admin browser/API behavior: NOT RUN
Current provider credential checks: NOT RUN
Current Wiki rows: NOT READ
Target release workflow: NOT IMPLEMENTED/ACTIVATED BY THIS RUNBOOK

---

Source: docs/runbooks/PLATFORM_TENANT_CONFIGURATION_ADMIN_SUPPORT_RUNBOOK.md — ctc repository · This page is an English translation of the source document. Identifiers, file paths and code symbols are unchanged.