Skip to main content

Multi-instance runtime

The Wrkbelt API runs as a horizontally-scaled cluster — multiple ECS tasks behind an ALB, sharing state through Redis, SQS, and MongoDB. Anything that mutates shared state has to coordinate; anything that fans out side-effects has to dedup. This page is the architectural reference: topology, primitives, and how they fit together. For the decision guide on which coordination primitive to pick for a new feature, see Cluster coordination pattern. For operational procedures — /health/ready triage, manual lock release, reading structured-log keys, capacity tripwires — see the Multi-instance runtime runbook.

Production topology​

Production autoscales between 1 and 3 tasks on CPU (primary signal, 60% target) and memory (co-signal, 80% target — sits above V8's typical 60–75% steady-state heap so GC laziness doesn't trip scale-out or block scale-in). See copilot/api/manifest.yml for the per-environment count blocks (develop inherits the default; staging pins min=max=1; production runs 1–3).

Deploys auto-rollback on the four cluster-coordination correctness alarms (redis.adapter.disconnected, redis.subscription.dropped, redis.lock.unavailable, redis.resource_lease.remove_failed) — deployment.rollback_alarms is wired per environment in the manifest, referencing the alarms defined in copilot/api/addons/observability.yml. A bad deploy that trips any of these reverts to the previous task definition without operator intervention.

The ALB uses sticky-session cookies (lb_cookie) so a single client lands consistently on one task — important for Socket.IO long-lived connections. Backend coordination assumes any task can handle any state-mutating request; sticky sessions are an optimization, not a correctness contract.

OpenSpec contracts: openspec/changes/add-multi-instance-runtime-readiness/proposal.md, openspec/changes/add-multi-instance-runtime-readiness/design.md, openspec/changes/add-multi-instance-runtime-readiness/specs/multi-instance-runtime/spec.md.

The four ioredis clients​

Each task owns four ioredis connections. They are separated by purpose, not for performance — pub/sub blocks the connection it uses, and Socket.IO's Redis adapter requires its own publisher and subscriber.

ClientPurposeSource
RedisCommandClientGeneral commands (GET/SET/DEL/SCAN, etc.)libs/api/services-api/src/lib/shared/redis/redis-connection.factory.ts
RedisAppSubscriberApplication pub/sub (e.g. ancestry-invalidate)same factory
RedisSocketIoPublisherSocket.IO Redis adapter — outboundsame factory
RedisSocketIoSubscriberSocket.IO Redis adapter — inboundsame factory

The factory and connection module live at libs/api/services-api/src/lib/shared/redis/redis.module.ts and redis-connection.factory.ts. Direct client injection is not allowed outside RedisModule and the Socket.IO adapter factory — see the boundary rule in Cluster coordination pattern.

The five semantic Redis services​

Application code consumes Redis through one of five semantic interfaces. Each owns its primitive's error policy, observability emits, and contract semantics. New code injects these — not the raw clients.

ServiceBackingContractSource
RedisCacheServiceRedis key/value + TTLRemoteCacheProvider (get/set/delete/getOrFetch)libs/api/services-api/src/lib/shared/redis/redis-cache.service.ts
RedisRateLimiterServiceAtomic counter + windowRateLimiter (acquire/tryAcquire)libs/api/services-api/src/lib/shared/redis/redis-rate-limiter.service.ts
RedisDistributedLockServiceSET NX EX + Lua compare-and-deleteDistributedLock (acquire/release/withLock)libs/api/services-api/src/lib/shared/redis/redis-distributed-lock.service.ts
RedisResourceLeaseServiceSame primitive as DistributedLock, richer RemoveResultResourceLease (claim/release)libs/api/services-api/src/lib/shared/redis/redis-resource-lease.service.ts
RedisBroadcastChannelServicePub/sub with auto-resubscribeBroadcastChannel (publish/subscribe)libs/api/services-api/src/lib/shared/redis/redis-broadcast-channel.service.ts

Cluster coordination pattern is the decision guide for picking among them. Don't reach for a sixth primitive without reading it first.

Socket.IO fanout​

Real-time scheduler events (the embedded widget) and operator-facing updates flow through Socket.IO. With multiple API tasks, a message published on task A must reach a client connected to task B. The Redis adapter solves this.

  • Adapter factory: libs/api/services-api/src/lib/core/socket/socket-redis-adapter.factory.ts — consumes the RedisSocketIoPublisher and RedisSocketIoSubscriber clients.
  • The factory is the only consumer of RedisSocketIoPublisher / RedisSocketIoSubscriber outside RedisModule.

When task A emits to a Socket.IO room, the adapter publishes on a Redis channel. All tasks subscribe; whichever task holds the matching client delivers the message.

Ancestry cache invalidation broadcast​

Org-ancestry data is cached per process with a 60-second TTL (Multi-tenant hierarchy → Ancestry cache). On every mutation that affects ancestry — membership change, role change, reparent — the org service publishes on the ancestry-invalidate Redis channel via RedisBroadcastChannelService. Every task subscribes and evicts the affected entry locally.

  • Source: libs/api/services-api/src/lib/core/organization/ancestry-cache.service.ts
  • Subscriber resilience: RedisBroadcastChannelService auto-resubscribes on TCP reconnect; gaps emit redis.subscription.dropped + redis.subscription.resubscribed with gap_ms for observability.

Graceful shutdown​

ECS sends SIGTERM to a task on deploy or scale-in. The API drains in-flight work cleanly:

  • HTTP server stops accepting new connections.
  • In-flight requests complete (bounded by ALB drain timeout).
  • SQS consumers drain their current batch and stop polling.
  • Cron locks held via RedisDistributedLockService are released; lock TTL is the safety net for non-graceful exits.
  • Socket.IO clients receive a transport close and reconnect to a surviving task.

Cron + SQS fanout​

The cluster has periodic scans that fanout per-record work (process pending media attachments, retry invoices, etc.). The pattern is cron-tickle + SQS fanout:

  1. A cron handler runs once per tick on exactly one task per (jobType, orgId) pair (gated by RedisDistributedLockService keyed on cron-fanout:<jobType>:<orgId>). The lock's TTL — derived from the cron interval — is the cooldown; the key itself carries no time component.
  2. The handler scans for eligible records and enqueues one SQS message per record.
  3. SQS FIFO with MessageDeduplicationId per record ensures at-most-once delivery per dedup window.
  4. The SQS consumer processes each record idempotently.

Source: libs/api/services-api/src/lib/scheduler/booking/booking.jobs-service.ts (the canonical reference implementation — tryEnqueueOrgJob).

For why this is a pattern rather than a sixth primitive, see Decisions → Deferred → Generic fanout abstraction.

Operational signals​

Every primitive emits structured-log keys consumed by CloudWatch metric filters and alarms. The contract is enforced by a test at libs/api/services-api/src/lib/shared/redis/observability-contract.spec.ts — adding a new redis.* key without an alarm or UNMONITORED_KEYS declaration fails CI. See the full alarm table in Cluster coordination pattern → Operational signals.

CloudWatch alarms ship in copilot/api/addons/observability.yml.