Cross-Region WebSocket State Sync #

Two users edit the same document. One is pinned to Ireland, the other to Sydney, and the room they share no longer exists in one place. Every message either crosses 250 ms of ocean or does not arrive at all. Then a WAN link stalls for forty seconds, recovers, and both sides replay their backlog into each other — producing a burst of stale cursor positions that users read as the document “rewinding”.

Replicating a real-time room between regions is three problems: getting messages across, stopping them coming back, and deciding what ordering you can honestly promise.

Root cause #

Within one region, fan-out is simple: every node subscribes to a shared bus, publishes locally, and delivers to its own sockets. Extending that across regions naively — pointing every region at one central bus — makes every message pay the WAN round trip, including the 95% of messages whose participants are all in one place.

The correct shape is a bus per region plus a replication bridge between them, and the bridge introduces three failure modes that do not exist in a single region:

Loops. Region A publishes, the bridge forwards to B, B’s bridge sees it on its local bus and forwards back to A. Without an origin marker this repeats until the link saturates, which takes seconds.

Replay storms. A partitioned link buffers messages; when it recovers, the entire backlog arrives at once. For state updates that backlog is not merely useless — it actively overwrites current values with older ones.

Ordering loss. Two regions publishing into one room have no shared clock and no shared sequence. There is no total order to recover, only per-origin order, and no amount of engineering inside the bridge changes that.

Loop prevention with an origin tag A message published in Europe carries an origin tag that the bridge preserves; when it appears on the Asia-Pacific bus the bridge recognises it as foreign and does not forward it back, breaking the loop. Loop prevention with an origin tag Node in EU EU bus AP bus Node in AP publish origin=eu bridge forwards (origin preserved) delivered to AP sockets bridge sees origin=eu — does not forward publish origin=ap bridge forwards Without the tag, each bridge re-forwards the other's messages forever and the link saturates in seconds
One field prevents the loop. Every other loop-prevention scheme is a more expensive version of this.

Resolution #

The bridge stays small: forward only what originated locally, drop anything stale, and republish remote traffic onto the local bus so ordinary fan-out delivers it.

import Redis from 'ioredis';

const REGION = process.env.REGION!;
const REPLICATION_TTL_MS = 5_000;      // older than this on arrival: drop it
const MAX_REPLICATED_BYTES = 32 * 1024;

interface Replicated {
  room: string;
  origin: string;                       // region that first published — the loop breaker
  seq: number;                          // per-origin monotonic
  ts: number;
  d: unknown;
}

const local = new Redis(process.env.REDIS_URL!);
const localSub = local.duplicate();
const peers = new Map<string, Redis>();     // one client per peer region
let seq = 0;

/** Local → peers. Only messages that originated here are forwarded. */
localSub.psubscribe('room:*');
localSub.on('pmessage', (_p, channel, raw) => {
  const msg = JSON.parse(raw) as Replicated;
  if (msg.origin !== REGION) return;               // already a replica: never echo
  if (raw.length > MAX_REPLICATED_BYTES) return;   // large payloads go via storage, not the WAN

  const envelope: Replicated = { ...msg, seq: ++seq, ts: Date.now() };
  const body = JSON.stringify(envelope);
  for (const [, client] of peers) {
    // Fire and forget: local delivery already happened, and blocking the local
    // fan-out on a WAN round trip would make every message pay for the crossing.
    client.publish(`replica:${channel}`, body).catch(() => {});
  }
});

/** Peers → local. Republish so normal per-node fan-out delivers it. */
export function subscribeToPeer(region: string, client: Redis): void {
  const sub = client.duplicate();
  sub.psubscribe('replica:room:*');
  sub.on('pmessage', (_p, channel, raw) => {
    const msg = JSON.parse(raw) as Replicated;
    if (msg.origin === REGION) return;              // our own message returning: drop

    // A recovered link delivers a backlog. Replaying forty seconds of stale
    // cursor positions is worse than dropping them — the next update is current.
    if (Date.now() - msg.ts > REPLICATION_TTL_MS) {
      metrics.replicationStaleDropped.inc({ from: region });
      return;
    }

    local.publish(channel.replace(/^replica:/, ''), JSON.stringify(msg));
  });
}

The TTL is the guard people leave out, and it is what turns a link recovery from an incident into a blip. Anything older than a few seconds is, for state traffic, strictly worse than nothing: the current value is already on its way.

Ordering needs an explicit decision rather than a mechanism. There are exactly two honest answers:

Per-origin ordering, converged state. Each region’s messages arrive in order relative to each other, and the client resolves conflicts with a per-entity version, as in handling out-of-order WebSocket messages. Correct for cursors, presence, chat and dashboards — anything where the newest value wins.

Home-region writes, total order. Each room is assigned an owning region; all writes route there, are sequenced there, and are replicated outward as an ordered stream. Writers outside the home region pay an extra round trip. Necessary for documents with operational transforms, auctions, ledgers — anything where the sequence of operations is the meaning.

// Home-region routing: writes go to the owner, reads are served locally everywhere.
function homeRegionFor(roomId: string): string {
  // Stable assignment so a room does not migrate on every deploy.
  const regions = ['eu-west-1', 'us-east-1', 'ap-southeast-2'];
  let hash = 0;
  for (let i = 0; i < roomId.length; i++) hash = (hash * 31 + roomId.charCodeAt(i)) | 0;
  return regions[Math.abs(hash) % regions.length];
}
What each ordering model gives you A comparison of per-origin ordering with versions, home-region writes, CRDTs and a single global region across write latency, ordering guarantees and the workloads each suits. What each ordering model gives you Write latency Ordering Good for Per-origin + versions local, fast per origin only cursors, presence, chat Home-region writes one WAN hop total documents, ledgers CRDT everywhere local, fast converges collaborative text Single global region worst case for most total small user base There is no row with local write latency and a global total order — that combination is not available over a WAN
Pick per room, not per system: cursors and the document they sit in can use different rows.

Verification #

Test the loop, the partition and the recovery — none of which occur by accident in staging.

# Loop check: publish once in one region, count deliveries everywhere
redis-cli -u "$REDIS_URL_EU" publish room:demo \
  '{"room":"demo","origin":"eu-west-1","ts":0,"seq":1,"d":1}'
redis-cli -u "$REDIS_URL_AP" --stat        # must show ONE delivery, not a rising rate

# Partition and recover: the backlog must be dropped, not replayed
sudo iptables -A OUTPUT -d "$AP_REDIS_IP" -j DROP && sleep 40
sudo iptables -D OUTPUT -d "$AP_REDIS_IP" -j DROP
curl -s localhost:9090/metrics | grep replication_stale_dropped_total

The health metric is the ratio of cross-region messages to total messages. In a well-shaped deployment it is small and stays flat as regions are added, because most rooms are local. If it grows with region count, your rooms are genuinely global and the home-region model will serve you better than replicating everything everywhere.

Watch replication_stale_dropped_total too. Zero means either the links are healthy or — more likely — the TTL guard is not wired in, and the next partition will replay its whole backlog into your users’ screens.

Cross-node messages per second as the fleet grows Cross-node message rate for 30 broadcasts per second: a peer mesh grows with n squared while a pub/sub bus grows linearly with node count. Cross-node messages per second as the fleet grows 120k sockets, 30 broadcasts/s — mesh vs one pub/sub bus Node-to-node mesh Redis pub/sub bus 0/s 500/s 1.0k/s 1.5k/s 2.0k/s 1 nodes 2 nodes 3 nodes 4 nodes 6 nodes 8 nodes
Cross-region traffic if every region forwards to every other: a full mesh grows with the square of the region count, which is why the bridge must forward only locally-originated messages.

Operational checklist #

FAQ #

Why not use one global Redis for every region? #

Because every publish then pays a WAN round trip, including messages whose sender and receivers are all in the same city. It also makes the whole system depend on one region’s availability — when that region has a problem, real-time stops everywhere rather than degrading in one place. A bus per region with replication keeps local traffic local and failures regional.

How do I stop the same message being delivered twice? #

The origin tag prevents loops, and a per-message id makes double delivery harmless if it happens anyway. The two together are what you want: the tag is the structural fix, and idempotent handling is the safety net, as covered in idempotent WebSocket message processing. Relying on the tag alone means one bridge bug becomes a data bug.

Can I get a global total order? #

Only by routing writes through a single sequencer, which is what the home-region model does per room. That is a real option and it is what collaborative editors with operational transforms need. What is not available is a global total order and local write latency everywhere — the network round trip is not something a design choice can remove.

What about presence across regions? #

Keep it per region and compute the union on read. Replicating every presence heartbeat globally turns your highest-frequency, cheapest traffic into WAN traffic for no benefit, since most rooms are single-region. Key presence on a stable session id so a user reconnecting into a different region does not appear twice — the deduplication described in fixing presence flapping and ghost users.

Should I use Redis Streams instead of pub/sub for the bridge? #

For state traffic, no — pub/sub is cheaper and the TTL guard already says you do not want a backlog. For traffic that must not be lost, yes: Streams give you replay and acknowledgement across the link, at the cost of storage and complexity. Many deployments run both, routing per message class, which is the same split described in Redis Streams vs Pub/Sub for WebSocket fan-out.

Back to Multi-Region & Edge WebSocket Delivery