Handling region failover for WebSockets #

Your EU region goes down at 14:02. Every WebSocket client there disconnects at once, retries, and — once DNS or the global load balancer steers them away — lands on the US region, which was sized for its own users and now receives 300,000 handshakes in two minutes. The US region’s accept queues overflow, its autoscaler lags, and its own users start dropping. Clients that do get through find no session state and resubscribe from scratch, hammering the databases with snapshot requests. A regional outage has become a global one. Failing WebSockets over between regions is harder than failing HTTP over, because every client is simultaneously disconnected and needs a new long-lived connection, and because the state those connections relied on lived in the failed region.

Root cause #

HTTP failover shifts new requests gradually: each request is independent, retries are cheap, and traffic moves as fast as clients make new requests. WebSocket failover shifts all connections at once. When a region fails, every client sees its connection drop within seconds and immediately tries to reconnect, producing a synchronized wave aimed at whatever the steering layer points to.

Three factors decide whether the surviving region absorbs it. Steering speed: DNS-based failover is bounded by TTLs and resolver caching, often minutes, during which clients retry against the dead region; anycast or a global load balancer reacts in seconds. Arrival shape: without jittered backoff, reconnects arrive in a spike rather than a ramp, and the peak second — not the total — is what overflows accept queues. State: sessions, presence and replay buffers held only in the failed region are gone, so every client must resync from snapshots, multiplying load on the surviving region’s data tier.

Reconnect arrivals at the surviving region Reconnect attempts per second for 300000 dropped clients: a fixed retry delay lands every client in one second, full jitter spreads them across 20 seconds. Reconnect arrivals at the surviving region 300k clients, 20s window — fixed delay vs full jitter Fixed delay Full jitter 0 100k 200k 300k 1s 2s 3s 4s 5s 6s 7s 8s 9s 10s 11s 12s 13s 14s 15s 16s 17s 18s 19s 20s
The surviving region must absorb the peak of this curve, not its average — jittered backoff with a wide ceiling is what flattens it.

Resolution #

Prepare four things before the outage, because none can be improvised during it.

1. Fast, health-driven steering. Put regions behind anycast or a global load balancer with health checks on each region’s readiness, so steering changes within seconds. If you rely on DNS, keep TTLs short (30–60 seconds) and remember that some resolvers ignore them.

2. Client-side spreading. Clients must reconnect with full-jitter backoff whose ceiling is large enough to spread a regional population over a minute or more, and must treat repeated failures to the same region as a signal to try the next endpoint rather than hammering the same one.

3. Capacity headroom. Each region needs spare connection and handshake capacity for its failover partner’s population, or an autoscaler fast enough to add it — which, for handshake-heavy spikes, usually means pre-provisioned headroom.

4. Replicated resume state. Sequence numbers and recent history for each stream must be available in the surviving region, so clients resume from their last sequence rather than requesting full snapshots.

// Client: multi-endpoint reconnect with wide jitter and region fallback.
const ENDPOINTS = ['wss://rt.example.com/ws', 'wss://rt-us.example.com/ws', 'wss://rt-eu.example.com/ws'];
const BASE_MS = 1_000;
const CAP_MS = 60_000;                // wide ceiling: spreads a regional herd over a minute
const FAILS_BEFORE_NEXT = 3;          // give the steered endpoint a few chances first

export class FailoverClient {
  private attempt = 0;
  private endpointIndex = 0;
  private failsOnEndpoint = 0;
  private lastSeq = new Map<string, number>();    // per stream, for resume anywhere

  constructor(private onMessage: (m: any) => void) {}

  connect() {
    const url = ENDPOINTS[this.endpointIndex];
    const ws = new WebSocket(url);
    ws.onopen = () => {
      this.attempt = 0;
      this.failsOnEndpoint = 0;
      // Resume every stream from its last sequence, in whichever region we reached.
      ws.send(JSON.stringify({ type: 'resume', streams: Object.fromEntries(this.lastSeq) }));
    };
    ws.onmessage = (e) => {
      const m = JSON.parse(e.data);
      if (m.stream && typeof m.seq === 'number') this.lastSeq.set(m.stream, m.seq);
      this.onMessage(m);
    };
    ws.onclose = () => {
      if (++this.failsOnEndpoint >= FAILS_BEFORE_NEXT) {
        this.endpointIndex = (this.endpointIndex + 1) % ENDPOINTS.length;   // try another entry point
        this.failsOnEndpoint = 0;
      }
      const cap = Math.min(CAP_MS, BASE_MS * 2 ** this.attempt++);
      setTimeout(() => this.connect(), Math.random() * cap);             // full jitter
    };
  }
}

The global endpoint (rt.example.com) comes first so steering normally decides; the regional endpoints are a fallback for when steering is slow or wrong. On the server side, admission control protects the surviving region while the wave arrives: per-node handshake budgets and readiness-based shedding from rate limiting WebSocket handshakes turn excess arrivals into quick 503s that clients back off from, instead of timeouts that tie up accept queues.

For resume to work in the other region, the replay buffer and sequence counters must replicate across regions — the concerns covered in cross-region WebSocket state sync. A few seconds of replication lag is acceptable: clients whose last sequence is ahead of the surviving region’s copy simply receive a snapshot.

A regional failover, prepared vs unprepared When the EU region fails, anycast or a global load balancer steers clients to the US within seconds, the jittered reconnect wave spreads over about a minute, and most clients resume from their sequence numbers by ninety seconds; a DNS-only setup may still point clients at the failed region five minutes later. A regional failover, prepared vs unprepared prepared: resumes under budget EU region fails (0 min) anycast/GLB steers to US (0.2 min) jittered wave, 1 min spread (0.5 min) most clients resumed from seq (1.5 min) DNS-only: TTL still pointing EU (5 min) Steering speed and client jitter decide the first minutes; replicated state decides the load after that
Failover quality is fixed long before the outage starts.

Edge cases #

Failing back. When the failed region recovers, do not flip all traffic back at once — that is a second storm. Shift steering weight gradually, and let the rebalancing mechanism from rebalancing WebSocket connections after scale-out move clients back over tens of minutes.

Partial failures. A region can be up for health checks but broken for WebSockets — its pub/sub layer down, say. Health checks should include the real-time path (readiness that checks pub/sub), or steering will keep sending clients to a region that accepts connections but delivers nothing.

Data residency. Failing over EU users to the US may violate residency requirements. Pair regions within the same jurisdiction, or fail over to a degraded mode (read-only, no history) rather than across borders.

Verification #

Rehearse failover as a game day, in staging first and then in production at low traffic. Mark one region unhealthy at the steering layer and measure: time until new connections land in the surviving region, the peak handshake rate there against its budget, the fraction of clients that resumed from a sequence versus fetched a snapshot, and user-facing error rates in both regions.

# During the drill: handshakes per second and rejections at the surviving region.
curl -s https://metrics.us.example.com/api/v1/query \
  --data-urlencode 'query=sum(rate(ws_upgrades_total[30s])) by (result)'
# Resume vs snapshot ratio after failover.
curl -s https://metrics.us.example.com/api/v1/query \
  --data-urlencode 'query=sum(increase(ws_resume_total[5m])) by (mode)'

A healthy drill shows the handshake peak under budget, most clients resuming rather than snapshotting, and no error-rate increase for users who were already in the surviving region.

Steering options for WebSocket failover DNS failover with a long TTL reacts in minutes with many retries against the dead region; short TTLs improve that; global load balancers and anycast react in seconds; a client-side endpoint list provides a backstop bounded by backoff. Steering options for WebSocket failover Reaction time Client retries to dead region Complexity DNS failover (TTL 300 s) minutes many low DNS failover (TTL 30 s) ~1 min some low Global load balancer seconds few managed Anycast seconds few higher Client endpoint list backoff-bound bounded in app Combine a fast steering layer with a client endpoint list as the backstop
Steering speed decides how many retries hit a dead region.

Operational checklist #

FAQ #

How do WebSocket clients fail over to another region? #

Their connections drop when the region fails; they reconnect with backoff, and a steering layer (anycast, a global load balancer or DNS) directs them to a healthy region, where they resume their streams from the last sequence number if state is replicated.

Why is WebSocket failover harder than HTTP failover? #

Because every connected client disconnects at the same moment and needs a new long-lived connection, so the surviving region receives a synchronized wave of handshakes. HTTP traffic shifts request by request and spreads naturally.

Is DNS failover good enough? #

It works, but it is slow: TTLs and resolver caching mean clients may keep trying the failed region for minutes. Use short TTLs, add a client-side endpoint list, or prefer anycast or a global load balancer for seconds-level reaction.

How much spare capacity does each region need? #

Enough to hold its failover partner’s peak connections and absorb its reconnect wave under the handshake budget. Many teams run active-active regions at 50–60% of capacity for exactly this reason.

Back to Multi-Region & Edge Delivery.