Multi-Region & Edge WebSocket Delivery #

Your service runs beautifully in eu-west-1. Then sales lands an account in Sydney, and the complaints start: typing feels sticky, cursors trail half a second behind, and the reconnect after a lift ride takes noticeably longer than it does in Berlin. Nothing is broken. Every packet from Sydney to Ireland crosses roughly 250 ms of round-trip latency, and a WebSocket connection pays that on the handshake, on every message, and — most painfully — on every reconnect, where TCP and TLS each cost their own round trips before the upgrade even starts.

Going multi-region for a request/response API is mostly a routing problem. Going multi-region for a stateful, long-lived socket is a state problem: two users editing the same document may be pinned to different continents, and the room they share does not exist in one place any more. This guide covers the three decisions that follow — how clients pick a region, how messages cross between regions, and what you give up on ordering when they do.

Prerequisites #

Single-region fan-out has to be solid first. You need the bus from Redis pub/sub fan-out working across nodes within one region, because cross-region replication is built on top of it rather than instead of it. You need message delivery guarantees settled — at-least-once with idempotent handling survives a cross-region hop; fire-and-forget with implicit ordering does not. And you need the connection-count metrics from WebSocket observability and monitoring tagged by region, or you will not be able to tell a routing bug from a demand shift.

Core implementation #

The replication bridge is deliberately small: it subscribes to the local bus, forwards messages that are not already replicas, and republishes remote messages locally with an origin tag so they cannot loop. Everything else — the socket nodes, the local fan-out — stays exactly as it was in a single region.

import Redis from 'ioredis';

const REGION = process.env.REGION!;                 // e.g. 'eu-west-1'
const PEERS = (process.env.PEER_REGIONS ?? '').split(',').filter(Boolean);
const REPLICATION_TTL_MS = 5000;                    // drop anything older than this

interface Replicated {
  room: string;
  origin: string;                                   // region that first published
  seq: number;                                      // per-region monotonic sequence
  ts: number;
  payload: unknown;
}

const local = new Redis(process.env.REDIS_URL!);
const localSub = local.duplicate();
const peers = new Map(PEERS.map((r) => [r, new Redis(process.env[`REDIS_URL_${r.toUpperCase().replace(/-/g, '_')}`]!)]));
const peerSubs = new Map([...peers].map(([r, c]) => [r, c.duplicate()]));

let seq = 0;

/** Local → remote: forward only messages that originated here. */
localSub.psubscribe('room:*');
localSub.on('pmessage', (_pattern, channel, raw) => {
  const msg = JSON.parse(raw) as Replicated;
  if (msg.origin !== REGION) return;                // it is already a replica: do not echo
  const envelope: Replicated = { ...msg, seq: ++seq };
  for (const [, client] of peers) {
    // Fire and forget across the WAN; the local delivery already happened.
    client.publish(`replica:${channel}`, JSON.stringify(envelope)).catch(() => {});
  }
});

/** Remote → local: republish onto the local bus so normal fan-out delivers it. */
for (const [region, sub] of peerSubs) {
  sub.psubscribe('replica:room:*');
  sub.on('pmessage', (_pattern, channel, raw) => {
    const msg = JSON.parse(raw) as Replicated;
    if (msg.origin === REGION) return;              // our own message came back: drop
    if (Date.now() - msg.ts > REPLICATION_TTL_MS) return;  // WAN stall: stale, drop it
    const localChannel = channel.replace(/^replica:/, '');
    local.publish(localChannel, JSON.stringify(msg));
  });
}

Two properties make this safe. The origin tag is the loop breaker — without it, two regions forward the same message to each other forever, and the failure mode is a bus that saturates in seconds. The timestamp check is the stall breaker: when a WAN link recovers after a partition, the queued backlog is worse than useless, because delivering thirty seconds of stale cursor positions produces a visible replay that users read as a bug.

Note what is not replicated: heartbeats, per-connection acks, and anything scoped to one socket. Replicating those multiplies WAN traffic by the number of regions for no benefit at all.

Evacuating a region without a stampede #

Multi-region only earns its cost if you can lose a region deliberately. That is not a routing exercise — DNS shifts in a minute — but a load exercise, because every socket that was terminated in the failed region reconnects at once into regions sized for their own traffic.

The arithmetic is unforgiving. A region holding 40,000 sockets that dies at once produces 40,000 TLS handshakes, 40,000 authentication checks, and 40,000 state resynchronisations, all landing inside whatever window your clients’ retry logic allows. With a fixed one-second retry, that entire load arrives in a single second: the receiving region sees a 40,000-connection spike, its CPU saturates on TLS, connections time out, and the clients that timed out retry — a self-sustaining storm that outlives the original failure by minutes.

Jitter is the whole fix, and it belongs on the client where the decision is actually made. The receiving region also needs headroom sized for the peak, not the average: with a twelve-second jitter window, the peak arrival rate is roughly the evacuated population divided by the window, so 40,000 clients need capacity for about 3,300 new connections per second on top of steady-state traffic. That number, not the total socket count, is what determines whether your failover is a non-event or an outage.

Three further details separate a clean evacuation from a messy one. First, drain rather than kill: closing sockets with 1001 Going Away tells clients this was intentional, and a client that distinguishes 1001 from 1006 can reconnect immediately on the first and back off on the second. Second, shed DNS before you shed connections, so new clients stop arriving at a region you are about to empty — otherwise you evacuate into your own inbound traffic. Third, pin reconnects briefly: a client that reconnects during an evacuation should be allowed to land anywhere, but a client reconnecting during a routine node restart should return to its own region so session resume still works. A short-lived region hint in the client’s stored session, honoured for about a minute, gets both cases right.

Rehearse it. A region evacuation that has never been tested is an assumption, and the number you are testing is settle time: how long from the first dropped socket until connection counts, error rates, and message latency are all back inside their normal bands. Under a minute is healthy. Several minutes means your jitter window is too narrow or your receiving capacity is too tight, and both are cheaper to fix on a Tuesday afternoon than during the real thing.

Configuration reference #

Edge cases & gotchas #

Ordering is per-region, not global. Once two regions publish into the same room, there is no total order without a coordinator. If your product needs one — a document with operational transforms, an auction, a ledger — designate a home region per room and route writes there, accepting the extra RTT for writers in exchange for correctness. If it does not — chat, presence, cursors — per-origin ordering plus a client-side sequence check is enough.

Presence double-counts across regions. Each region’s presence set knows its own members. A naive union counts a user who reconnected from Ireland to Sydney twice until the first TTL expires, which users see as their own ghost. Key presence on a stable session id rather than a connection id, and let the newer entry win.

Failover moves connections, not state. When a region evacuates, tens of thousands of clients reconnect simultaneously into a region that was sized for its own load. Jittered reconnect is mandatory rather than nice-to-have here — see exponential backoff with jitter for WebSocket reconnects — and the receiving region needs headroom for the surge before you rely on the failover at all.

Edge termination is not edge state. Terminating TLS and the WebSocket upgrade at the edge cuts handshake latency substantially, because the expensive round trips happen close to the user. It does not move your room state, so every message still crosses to the origin. The win is real but bounded: fast connects, unchanged message latency, unless the edge platform gives you stateful objects that can host the room itself.

Verification #

Measure from where your users are, not from your laptop. Synthetic probes in each region that open a socket, round-trip a message, and record the split between connect time and message time will tell you within a day whether routing is doing what you think.

# Which region does DNS actually hand out from here?
dig +short ws.example.com @resolver1.opendns.com

# Connect and time the handshake separately from the first message
time npx wscat -c wss://ws.example.com/socket --no-color -x '{"t":"ping"}'

# Replication loop check: publish once in one region, count deliveries in both
redis-cli -u "$REDIS_URL_EU" publish room:demo '{"room":"demo","origin":"eu-west-1","ts":0,"payload":1}'
redis-cli -u "$REDIS_URL_AP" --stat

The metric that proves the design works is the ratio of cross-region messages to total messages. In a healthy deployment it is small — most rooms are local — and it stays flat as you add regions. If it climbs with region count, your rooms are genuinely global and you should be pinning them to a home region rather than replicating everything everywhere.

Guides in this area #

• Routing Clients to the Nearest WebSocket Region — latency-based DNS versus anycast, the reconnect pin that keeps sessions resumable, and how to evacuate a region without a stampede.
• Cross-Region WebSocket State Sync — the replication bridge in full, loop prevention, ordering guarantees you can and cannot offer, and presence deduplication across regions.

FAQ #

Do I actually need multiple regions, or just a CDN? #

A CDN does nothing for a WebSocket beyond the initial HTTP handshake, because there is no cacheable response — the connection is the payload. What helps is terminating the connection closer to the user, which requires either regional deployments or an edge platform that speaks WebSocket. If your users are within about 100 ms of your single region, invest in message efficiency first; the gains are larger and the cost is far lower.

How do two users in different regions edit the same document? #

Either you accept eventual consistency and use a data structure that converges regardless of order — a CRDT — or you pick a home region per document and route all writes through it. There is no third option that preserves both low write latency everywhere and a single total order; that trade is a property of the network, not of your framework.

What changes for Socket.IO versus raw ws? #

Socket.IO’s Redis adapter already does node-to-node fan-out, and there are cluster adapters that extend it across regions. The loop-prevention and TTL concerns above still apply, and the adapter’s broadcast semantics assume a single logical bus — so a WAN partition surfaces as silently divided rooms rather than an error. On raw ws you write the bridge and therefore control what happens when the link degrades.

Does sticky routing still matter with multiple regions? #

Yes, and at two levels. Within a region you still need the node-level stickiness described in load balancer sticky sessions if any per-connection state lives in process. Across regions you need session-level stickiness for long enough that a reconnecting client can resume rather than resynchronise, which is why the reconnect pin exists in the table above.

How do I test a region failure before it happens? #

Evacuate a region on purpose during a low-traffic window: fail its health check, watch DNS shift, and measure how long the reconnect surge takes to settle and what the receiving region’s connection count peaks at. The number you are looking for is the settle time — if it exceeds a minute, your backoff needs more jitter or your receiving region needs more headroom.

Where should presence state live in a multi-region deployment? #

Locally in each region, with a cross-region union computed only when a room actually spans regions. Replicating every presence heartbeat globally turns the cheapest, highest-frequency traffic you have into WAN traffic, and it buys nothing for the overwhelming majority of rooms whose members are all in one place. Publish per-region presence summaries on a slower cadence — every few seconds rather than every heartbeat — and merge them on read.

Does an edge platform remove the need for regions? #

It changes the shape of the problem rather than removing it. Terminating the socket at the edge cuts connection setup dramatically, because the TLS and upgrade round trips happen tens of milliseconds from the user instead of hundreds. But unless the edge platform can host your room state as well — stateful objects addressable by room id, rather than stateless functions — every message still travels to an origin, and message latency is unchanged. Check which of the two you are buying before assuming the lag problem is solved.

How many regions should I actually run? #

Fewer than you think. Each region multiplies the replication matrix, the deployment surface, and the number of ways ordering can surprise you, while the latency benefit falls off sharply after the first two or three well-chosen locations. Start with one region per continent where you have meaningful traffic, measure the p95 connect and round-trip times your real users see, and add a region only when a specific population is demonstrably outside the band you promised them.

Related #

• Redis Pub/Sub Fan-Out for WebSockets — the in-region bus this replication bridge sits on top of.
• Presence & Online Tracking — presence sets that need session-keyed deduplication once they span regions.
• Message Delivery Guarantees — the at-least-once semantics that make a cross-region hop survivable.
• Horizontal Scaling on Kubernetes — per-region capacity and the headroom a failover surge demands.
• Load Balancer Sticky Sessions — node-level affinity that remains necessary inside each region.

Back to Scaling Real-Time Infrastructure