Multi-instance runtime runbook

Operational procedures for the autoscaled Wrkbelt API cluster — what to do, in what order, when something looks wrong in production. For the architectural reference (production topology, the four ioredis clients, semantic Redis services, Socket.IO fanout, ancestry-invalidate broadcast, the three-timer drain ladder design) see Architecture → Multi-instance runtime.

Verify environment and dependency state

Three endpoints, three contracts:

• curl https://api.wrkbelt.com/info → { environment, uptime_seconds }. environment reads from getEnv().ENVIRONMENT (the canonical accessor — NODE_ENV is hard-coded production for all deployed builds in the Dockerfile and would misreport develop/staging). Use this to confirm you're hitting the env you think you are.
• curl https://api.wrkbelt.com/health/ready → terminus body { status, info, error, details } with mongo + redis indicators. HTTP 200 if all up, 503 if any down. The ALB does not consume this endpoint — it reads /health/live only — so a 503 here does not drop the task from rotation. It's an observability probe.
• curl https://api.wrkbelt.com/health/live → always 200 if the process can serve HTTP. This is what the ALB target group reads.

The fact that /info responds at all is the smoke check that Redis-backed cluster coordination is wired: under fail-loud startup (apps/api/src/main.ts), the process cannot boot without UPSTASH_REDIS_TCP_URL. A reachable /info proves the four ioredis clients (command, app-subscriber, Socket.IO publisher, Socket.IO subscriber) are live.

/health/ready returns 503

The terminus body tells you which indicator is down:

{
  "status": "error",
  "info": { "mongo": { "status": "up" } },
  "error": { "redis": { "status": "down" } },
  "details": { ... }
}

Escalation:

1. Identify the down indicator (error.<name>.status: 'down').
2. Check the corresponding CloudWatch alarms (copilot/api/addons/observability.yml):
   • redis.adapter.disconnected — Socket.IO Redis adapter dropped TCP.
   • redis.subscription.dropped — app pub/sub subscriber dropped TCP.
   • redis.lock.unavailable — distributed-lock service cannot reach Redis.
3. Check the Upstash console for outage indicators.
4. Rollback only if the regression is deploy-correlated. A 503 during a deploy bake window indicates the new task definition can't reach Redis (config drift, env-var corruption); a 503 outside a deploy window usually indicates an Upstash blip that will recover when Upstash does. The deploy circuit breaker (deployment.rollback_alarms in copilot/api/manifest.yml) auto-rolls back on the four cluster-coordination correctness alarms — no manual rollback is required during a deploy.

Manual DistributedLock release

Cron-fanout locks follow the key pattern cron-fanout:<jobType>:<orgId> — e.g., cron-fanout:MEDIA_PROCESS:6841234567890abcdef12345. TTL is the cron interval × 0.9 (a 5-minute media-process cron holds for 4m30s), so locks self-expire. Manual release is only needed if a sustained Redis outage caused a release to silently fail and the stale lock blocks the next tick.

# Find cron-fanout locks
redis-cli --tls -u "$UPSTASH_REDIS_TCP_URL" KEYS 'cron-fanout:*'

# Inspect the holder (returns the holder's randomUUID() token)
redis-cli --tls -u "$UPSTASH_REDIS_TCP_URL" GET 'cron-fanout:MEDIA_PROCESS:6841234567890abcdef12345'

# Check TTL — if positive, leave it alone; the lock will self-expire
redis-cli --tls -u "$UPSTASH_REDIS_TCP_URL" TTL 'cron-fanout:MEDIA_PROCESS:6841234567890abcdef12345'

# Release (only after confirming the holding instance is no longer running)
redis-cli --tls -u "$UPSTASH_REDIS_TCP_URL" DEL 'cron-fanout:MEDIA_PROCESS:6841234567890abcdef12345'

KEYS * is not used in application code (it's banned by the cluster-coordination pattern), but is safe in operator-driven incident response — the production keyspace is small enough that KEYS returns quickly. Never run FLUSHDB / FLUSHALL / SCRIPT FLUSH against this connection: the dev/staging/production Upstash database is shared (per-environment isolation is via keyPrefix, which database-level commands bypass).

Drain timing on deploy

Graceful shutdown is governed by a three-timer ladder:

Tier	Setting	Source	Purpose
In-app per-arm	10s (GRACEFUL_DRAIN_TIMEOUT_MS)	libs/api/utils-api/src/lib/lifecycle/	HTTP, SQS, Socket.IO, and Redis-quit arms run concurrently inside OnModuleDestroy / OnApplicationShutdown; 10s ceiling on each
ALB deregistration	60s (http.deregistration_delay)	copilot/api/manifest.yml	50s headroom over the in-app ceiling; AWS default is 300s (overridden here)
ECS hard kill	90s (ContainerDefinitions[0].StopTimeout via taskdef_overrides)	copilot/api/manifest.yml	30s buffer over ALB drain; long-blocked ServiceTitan calls SIGKILL at this ceiling

A typical deploy: ECS sends SIGTERM → in-app drain completes within ~10s → ALB finishes draining at the 60s mark → ECS stops the task. A stalled ServiceTitan call inside BaseServiceTitanClient.makeRequest (no fetch timeout configured) hits SIGKILL at the 90s ceiling.

Reading structured-log keys in CloudWatch

Cluster-coordination signals emit as structured-log lines (<key> <JSON payload>) on the task's stdout, shipped to CloudWatch Container Insights. Metric filters at copilot/api/addons/observability.yml anchor on the trailing space after the key to rule out suffix collisions.

Key	When	Alarm	Threshold (5min Sum)
redis.adapter.disconnected	Socket.IO Redis adapter dropped TCP mid-flight	Yes	≥ 1
redis.adapter.reconnected	Adapter subscriber reconnected; carries gap_ms	No	informational pair
redis.subscription.dropped	App-subscriber (e.g., ancestry-invalidate) dropped TCP	Yes	≥ 3
redis.subscription.resubscribed	App-subscriber reconnected; carries gap_ms	No	informational pair
redis.lock.unavailable	DistributedLock.acquire could not reach Redis	Yes	≥ 5
redis.resource_lease.reclaim_detected	Cross-instance lease handoff observed (sibling reclaimed a session)	No	informational
redis.resource_lease.remove_failed	Lease release threw on TCP loss — BOOKING_ABANDONED silently suppressed	Yes	≥ 1
redis.cron_fanout.enqueue_failed	Cron handler couldn't enqueue an SQS message after holding the lock	Yes	sustained

CloudWatch Logs Insights query for any redis.* event in the last 30 minutes:

fields @timestamp, @message
| filter @message like /redis\./
| sort @timestamp desc
| limit 100

Metric filters publish into Wrkbelt/Api/<EnvCapitalized> (Wrkbelt/Api/Production, Wrkbelt/Api/Staging). Alarms ship in staging + production only (gated by the IsAlarmEnv condition in addons/observability.yml); develop emits metrics but no alarms — dev traffic is too noisy to threshold meaningfully and a dev outage is not actionable.

deployment.rollback_alarms in copilot/api/manifest.yml ties the four correctness alarms (redis-adapter-disconnected-sustained, redis-subscription-dropped-sustained, redis-lock-unavailable-sustained, redis-resource-lease-remove-failed) to ECS deployment health: a bad deploy that trips any of them reverts to the previous task definition without operator intervention.

Capacity tripwires

The architecture's first response to a capacity concern is to upgrade the Upstash plan — not to migrate to a different store. The semantic-interface architecture (RemoteCacheProvider, RateLimiter, DistributedLock, ResourceLease, BroadcastChannel) makes a future store-swap implementation-internal, but the current posture is managed Upstash.

Sustained breach of any of the following over the stated window warrants explicit re-evaluation:

1. Broadcast latency SLO — p99 cross-instance broadcast latency > 2s sustained for 7 days, OR p95 > 500ms sustained for 7 days.
2. Plan throttling — Upstash dashboard reports throttled requests > 1% of total commands sustained for 7 days.
3. Pub/sub QPS saturation — sustained pub/sub QPS > 50% of plan ceiling for 30 days.
4. Cost crossover (informational) — managed-plan cost materially exceeds an equivalent self-managed alternative at our scale.
5. Latency-sensitive new feature — a new use case requires sub-millisecond Redis read latency that public-internet Upstash TCP cannot deliver.

The evaluation may conclude "upgrade plan," "switch tier," or, in the limit, "migrate to a different store." The answer is not predetermined.

Related

• Architecture → Multi-instance runtime — production topology, the four ioredis clients, semantic Redis services, Socket.IO fanout, ancestry-invalidate broadcast
• Architecture → Cluster coordination pattern — decision guide for picking the right cross-instance coordination primitive
• CloudWatch alarms: copilot/api/addons/observability.yml
• Per-environment count blocks and rollback alarms: copilot/api/manifest.yml