Source document · API/docs/OPERATIONS.md

Production operations runbook

Readiness

Set DATABASE_URL and run go run ./cmd/migrate before deployment. /ready is the deployment gate; a production deployment must also verify PostgreSQL with pg_isready and an authenticated migration check.

Staging and production startup require DATABASE_URL, AUTH_TOKEN_SECRET (at least 32 bytes), an email provider, HTTPS S3-compatible storage credentials, and LiveKit wss:// URL/API key/API secret. For MailAfrica, set MAILAFRICA_API_KEY and MAILAFRICA_FROM_ADDRESS; the key must be the server-side MAIL_ key from the MailAfrica dashboard. To send from your own verified domain, also set MAILAFRICA_FROM_DOMAIN_ID to its numeric domain ID and use an address registered for that domain. Without a domain ID, MailAfrica requires a permitted address on its platform sender domain. The API sends transactional verification, sign-in, and password-reset messages through POST https://api.mailafrica.online/api/outbound/emails. MailAfrica currently documents a charge of 5 TZS per recipient, debited from its wallet, and a limit of two individual sends per second; keep the wallet funded. If no MailAfrica key is configured, SMTP remains available and must offer STARTTLS. They also require LISTING_SUBSCRIPTION_AMOUNT in currency minor units, LISTING_SUBSCRIPTION_CURRENCY, and LISTING_SUBSCRIPTION_DAYS; these values determine subscription payment amounts and renewal extensions on the server. Manual WeChat and Alipay payment collection also requires WECHAT_PAY_INSTRUCTIONS and ALIPAY_PAY_INSTRUCTIONS. Set these to the verified merchant account/QR details and the steps customers must follow. The API includes the method-specific text in payment responses; it never invents a recipient. Keep the values current with the merchant accounts actually able to receive payments. Only development, test, staging, and production are accepted as APP_ENV; unknown values fail startup so a typo cannot select in-memory repositories. /ready is enabled automatically in staging and production and returns 503 when PostgreSQL cannot be reached. Keep migration execution in the release pipeline before starting the new application version.

Backup and restore

Use managed PostgreSQL point-in-time recovery where available. Before a release, record the backup identifier. For a restore drill, restore into an isolated database, run migrations, execute DATABASE_URL=... go test ./..., and verify order/payment/community/reporting row counts before promotion.

Load and security validation

Run authenticated API load tests against a staging database only. Confirm rate-limit responses, request-size rejection, token expiry, role boundaries, SQL-injection payloads, private-media authorization, and webhook signature validation. Never run destructive restore or load tests against production.

Observability

Scrape /metrics and correlate request_id values from structured logs. Alert on readiness failures, elevated 5xx responses, rate-limit spikes, database connection exhaustion, and payment webhook backlog.

Outbox notifications

Order-created events are delivered to customer and provider in-app inboxes by a worker running in the API process. The outbox is at-least-once; notification IDs are deterministic so retrying a partially delivered event does not duplicate an inbox item. Failed events back off for up to 256 seconds and are dead-lettered after 10 attempts. Check for exhausted events with:

SELECT id, event_type, aggregate_id, attempts, last_error, dead_lettered_at
FROM outbox_events
WHERE dead_lettered_at IS NOT NULL
ORDER BY dead_lettered_at DESC;

Correct the underlying data or delivery issue before manually clearing the dead-letter marker and setting available_at = NOW() to retry an event.

Android push delivery

Configure FIREBASE_PROJECT_ID and FIREBASE_CREDENTIALS_FILE with a Firebase service account authorized for Firebase Cloud Messaging. The service account file must be mounted outside the repository and kept private. When configured, persisted order notifications are also sent to enabled Android device tokens. Without these values, notifications remain available in the authenticated in-app inbox.