Production Environment

• Status: CURRENT RUNBOOK
• Owns: approved production environment configuration and read-only inspection procedure
• Does not own: deployment authorization, current Production state proof or application secrets
• Update when: approved service, environment, credential-installation or inspection procedure changes

This document captures the production runtime environment contract for CTC. Keep application code in Git and keep production secrets outside the Git worktree.

PHP-FPM Environment Variables

Production runtime variables are managed in:

/etc/ctc/prod.env

php-fpm loads that file through the systemd drop-in:

/etc/systemd/system/php-fpm.service.d/override.conf

The drop-in should use:

[Service]
EnvironmentFile=/etc/ctc/prod.env

Restart=always
RestartSec=5

Do not add new per-key Environment= entries to the php-fpm systemd drop-in.

/etc/php-fpm.d/www.conf should keep:

clear_env = no

With clear_env = no, PHP can read variables loaded by systemd through getenv() without per-key env[KEY] = $KEY pass-through lines.

Updating Production Variables

Edit only the external env file:

sudo vi /etc/ctc/prod.env

Use quoted values to avoid parsing issues:

DB_HOST="db.cwto1o3arnd0.us-west-2.rds.amazonaws.com"
DB_NAME="prod"
DB_PORT="3306"
DB_USER="ctc"
DB_PASS="..."
TELEGRAM_TOKEN="..."
CTC_LOG_ROOT="/var/log/ctc"
GMAIL_CREDENTIALS_PATH="/etc/ctc/gmail/credentials.json"
GMAIL_ADDON_TOKEN="<private value in .agents/rules/DEV_ENV.md>"

After editing, validate and restart:

sudo systemctl daemon-reload
sudo php-fpm -t
sudo systemctl restart php-fpm

PHP-FPM does not reload process environment variables from the file until the service is restarted.

Permissions

Keep the env file root-owned and readable only by root:

sudo chown root:root /etc/ctc/prod.env
sudo chmod 600 /etc/ctc/prod.env

Do not commit production .env files or secrets into the repository.

Git Autopull

Production may periodically sync the worktree with:

git fetch --prune origin
git reset --hard origin/master

Because /etc/ctc/prod.env is outside /var/www/html, it is not affected by git reset --hard.

Runtime-generated files, uploads, logs, and production-only config should also stay outside Git-managed code paths unless the app explicitly owns them.

Gmail OAuth Credentials

Gmail OAuth client credentials must live outside the Git worktree. The app reads GMAIL_CREDENTIALS_PATH, then GOOGLE_APPLICATION_CREDENTIALS, then checks the external default:

/etc/ctc/gmail/credentials.json

Preferred installation from the local workstation:

./scripts/install-gmail-credentials.sh /absolute/path/to/rotated-google-oauth-client.json ctc

The source JSON must be the newly rotated OAuth client JSON from Google Cloud. Do not use a file from this repository or from /var/www/html/config/credentials.json.

The helper validates the OAuth client shape before and after install. The JSON must include web.client_id, web.client_secret, web.auth_uri, web.token_uri, and web.redirect_uris containing https://ctclogistic.com/public/gmail/callback.php. Override the expected callback only when deploying a different host:

EXPECTED_GMAIL_REDIRECT_URI="https://example.com/public/gmail/callback.php" \
  ./scripts/install-gmail-credentials.sh /absolute/path/to/google-oauth-client.json ctc

Manual production setup, if the helper script is not available:

sudo install -d -m 0750 -o root -g apache /etc/ctc
sudo install -d -m 0750 -o root -g apache /etc/ctc/gmail
sudo install -m 0640 -o root -g apache /path/to/google-oauth-client.json /etc/ctc/gmail/credentials.json
sudo -u apache test -r /etc/ctc/gmail/credentials.json
sudo -u apache php -r '$p="/etc/ctc/gmail/credentials.json"; $expected="https://ctclogistic.com/public/gmail/callback.php"; $json=file_get_contents($p); $data=is_string($json)?json_decode($json,true):null; $web=is_array($data)&&isset($data["web"])&&is_array($data["web"])?$data["web"]:null; $missing=array(); foreach(array("client_id","client_secret","auth_uri","token_uri") as $key){ if(!is_array($web)||!isset($web[$key])||!is_string($web[$key])||trim($web[$key])===""){ $missing[]="web.".$key; }} if(!is_array($web)||!isset($web["redirect_uris"])||!is_array($web["redirect_uris"])||count($web["redirect_uris"])===0){ $missing[]="web.redirect_uris"; } elseif(!in_array($expected,$web["redirect_uris"],true)){ $missing[]="web.redirect_uris expected callback"; } if(is_array($web)&&isset($web["client_id"])&&is_string($web["client_id"])&&substr($web["client_id"],-27)!==".apps.googleusercontent.com"){ $missing[]="web.client_id Google OAuth suffix"; } if(count($missing)>0){ fwrite(STDERR,"Invalid Gmail OAuth credential JSON: ".implode(", ",$missing).PHP_EOL); exit(1); }'
sudo vi /etc/ctc/prod.env
sudo chown root:root /etc/ctc/prod.env
sudo chmod 600 /etc/ctc/prod.env
sudo php-fpm -t
sudo systemctl restart php-fpm

The env file should contain:

GMAIL_CREDENTIALS_PATH="/etc/ctc/gmail/credentials.json"
GMAIL_ADDON_TOKEN="<private value in .agents/rules/DEV_ENV.md>"

Gmail Workspace Add-on Gateway

The Gmail add-on calls the public appointment gateway:

/public/gmail/addon.php

Configure a dedicated shared secret for Apps Script. Do not reuse the Gmail controller or operation token:

GMAIL_ADDON_TOKEN="<private value in .agents/rules/DEV_ENV.md>"
GMAIL_ADDON_SYSTEM_BASE_URL="https://ctclogistic.com"
GMAIL_ADDON_REVIEW_PATH="/transload-email-approval"

Set the matching Apps Script properties on the Gmail Workspace Add-on project:

GMAIL_ADDON_TOKEN=<same shared secret>
GMAIL_ADDON_API_URL=https://ctclogistic.com/public/gmail/addon.php

GMAIL_ADDON_TOKEN is used only for the Gmail Workspace Add-on gateway at /public/gmail/addon.php. Store the same value in the Apps Script project's script properties. GMAIL_ADDON_SYSTEM_BASE_URL and GMAIL_ADDON_REVIEW_PATH are optional when the defaults are correct. The Apps Script request must send Authorization: Bearer <GMAIL_ADDON_TOKEN> or X-CTC-Gmail-Addon-Token.

Do not restore config/credentials.json in the repository. A production deploy may run git reset --hard origin/master, which intentionally removes repo-local credentials.

If a repo-local config/credentials.json ever existed in production or Git history, treat that OAuth client secret as exposed. Rotate the Google OAuth client secret first, then install the new JSON at /etc/ctc/gmail/credentials.json. Do not copy the exposed file from /var/www/html/config/credentials.json into /etc/ctc/gmail/credentials.json.

Runtime Logs

JSONL runtime logs are written through JsonlLogger.

• Production should set CTC_LOG_ROOT="/var/log/ctc" in /etc/ctc/prod.env.
• The log root must be writable by the PHP-FPM runtime user.
• If CTC_LOG_ROOT is not set, the app falls back to storage/logs under the web root.
• Do not point runtime logs at media/logs; media may be backed by EFS or upload storage and is not the durable log contract.

Expected production setup:

sudo install -d -m 0775 -o apache -g apache /var/log/ctc
sudo systemctl restart php-fpm

SSH, SFTP, And DB Access

Local operator SSH config should provide this alias:

ssh ctc
sftp ctc

Expected SSH target:

Host ctc
HostName ec2-34-220-102-225.us-west-2.compute.amazonaws.com
Port 2222
User ec2-user
IdentityFile ~/.ssh/ctc250616.pem

Quick SSH verification:

ssh ctc 'hostname && whoami && pwd'

Expected verified state on 2026-05-12:

• SSH user: ec2-user
• Remote home: /home/ec2-user
• Web root: /var/www/html
• Production env file: /etc/ctc/prod.env

SFTP verification:

printf 'pwd\nbye\n' | sftp -b - ctc

Expected remote working directory:

/home/ec2-user

For production MySQL read checks from the local workstation, use the canonical SSH path below. Do not assume the optional ctc-db helper is installed; check with command -v ctc-db before using it. A missing helper is not a production DB access failure.

ssh ctc 'sudo -n bash -s' <<'REMOTE'
set -euo pipefail
set -a
source /etc/ctc/prod.env
set +a
MYSQL_PWD="$DB_PASS" mysql --protocol=TCP -h "$DB_HOST" -P "$DB_PORT" -u "$DB_USER" --batch "$DB_NAME" \
  -e 'SELECT @@hostname AS db_host, @@port AS db_port, DATABASE() AS db_name, CURRENT_USER() AS db_user, VERSION() AS version;'
REMOTE

/etc/ctc/prod.env is root-owned by design, so normal-user read failure is expected. The query must run through ssh ctc and sudo -n; do not use ssh hub or the repository-local .env, which targets the local DB mirror. Always run and inspect the identity query before a ticket/data query.

Read-only DB smoke test:

Use the same canonical path with both read-only queries:

ssh ctc 'sudo -n bash -s' <<'REMOTE'
set -euo pipefail
set -a
source /etc/ctc/prod.env
set +a
MYSQL_PWD="$DB_PASS" mysql --protocol=TCP -h "$DB_HOST" -P "$DB_PORT" -u "$DB_USER" --batch "$DB_NAME" <<'SQL'
SELECT @@hostname AS db_host, @@port AS db_port, DATABASE() AS db_name, CURRENT_USER() AS db_user, VERSION() AS version;
SELECT COUNT(*) AS table_count FROM information_schema.tables WHERE table_schema = DATABASE();
SQL
REMOTE

Verified result on 2026-05-12:

db    user   version
prod  ctc@%  8.0.45

table_count
59

Do not run production writes through this path unless the user explicitly confirms the exact write operation.

Verification

Use these checks after an environment change:

sudo stat -c "%U:%G %a %n" /etc/ctc/prod.env
sudo stat -c "%U:%G %a %n" /etc/ctc/gmail/credentials.json
sudo systemctl cat php-fpm
sudo grep -n "^[[:space:]]*clear_env\|^[[:space:]]*env\[" /etc/php-fpm.d/www.conf
curl -I http://localhost/login.php

Expected state:

• /etc/ctc/prod.env is root:root and mode 600.
• /etc/ctc/gmail/credentials.json is outside /var/www/html, readable by the PHP-FPM runtime group, and not committed to Git.
• php-fpm uses EnvironmentFile=/etc/ctc/prod.env.
• www.conf has clear_env = no.
• www.conf does not need per-key env[KEY] = $KEY lines.
• login.php returns HTTP 200 locally.

Deployment Scope

CTC deployment is triggered by the repository's master push. Do not run a deployed-SHA confirmation, autopull.timer inspection, or service preflight as part of a normal deployment. Follow the repository deployment rules and run only the minimum post-deploy health check they require.

---

Source: docs/runbooks/PRODUCTION_ENVIRONMENT_RUNBOOK.md — ctc repository · This page is an English translation of the source document. Identifiers, file paths and code symbols are unchanged.