Routing WebSocket Clients to the Nearest Region #

You deploy a second region in Sydney and Australian users still connect to Ireland. DNS says it is routing by latency, the health checks are green, and the round-trip time in your dashboard has not moved. The cause is almost always the same: DNS routes the resolver, not the user, and a corporate or public resolver in a third location makes that decision on everyone’s behalf.

Routing long-lived connections has properties that HTTP routing does not, and the differences decide which mechanism you should use.

Root cause #

Three mechanisms are available, and each fails differently for WebSockets.

Latency-based DNS. The provider measures round-trip times from resolvers to regions and returns the closest. It works well when users use their ISP’s resolver and poorly when they use a centralised public resolver — with EDNS Client Subnet the provider gets a hint about the real client network, but not every resolver sends it and not every provider honours it. The other limitation is that DNS decides only at connect time; a connection already established is unaffected by any later change.

Anycast. The same IP is announced from multiple locations and BGP routes each packet to the topologically nearest one. Routing is fast and needs no DNS TTL, but BGP can re-converge mid-session and move your packets to a different site — which for a stateless UDP service is fine and for a TCP connection means the connection dies. Anycast for WebSockets requires that any site can serve any connection, or that re-convergence is rare enough to treat as a fault.

Client-side probing. The client measures actual connect latency to each regional endpoint and picks the fastest. It is the only mechanism that measures what you actually care about — the user’s real round trip to the real endpoint — but it costs a probe round trip on first load and needs somewhere to cache the result.

For a WebSocket the routing decision is made once and lives for hours, so the cost of measuring properly is amortised over a long session. That tilts the answer towards probing more than it would for HTTP.

Resolution #

Use DNS as the default and let the client correct it. A short probe on first load, cached for a day, gets the accuracy of measurement without paying for it on every connection.

const REGIONS = [
  { id: 'eu-west-1',      url: 'wss://eu.ws.example.com/ws' },
  { id: 'us-east-1',      url: 'wss://us.ws.example.com/ws' },
  { id: 'ap-southeast-2', url: 'wss://ap.ws.example.com/ws' },
];

const PROBE_TIMEOUT_MS = 2_500;
const PIN_TTL_MS = 24 * 60 * 60 * 1000;      // re-probe daily
const RECONNECT_PIN_MS = 60_000;             // stay put across a brief reconnect

interface Pin { regionId: string; url: string; chosenAt: number; }

function readPin(): Pin | null {
  try {
    const raw = localStorage.getItem('ws.region');
    if (!raw) return null;
    const pin = JSON.parse(raw) as Pin;
    return Date.now() - pin.chosenAt < PIN_TTL_MS ? pin : null;
  } catch { return null; }
}

/** Measure real connect time to each region; the winner is the one that answers first. */
async function probe(): Promise<Pin> {
  const attempts = REGIONS.map((region) => new Promise<Pin>((resolve, reject) => {
    const started = performance.now();
    // A tiny /probe endpoint that accepts the upgrade and closes immediately —
    // this measures TCP + TLS + upgrade, which is exactly what a real connect costs.
    const socket = new WebSocket(region.url.replace(/\/ws$/, '/probe'));
    const timer = setTimeout(() => { socket.close(); reject(new Error('timeout')); }, PROBE_TIMEOUT_MS);
    socket.onopen = () => {
      clearTimeout(timer);
      socket.close(1000, 'probe');
      resolve({ regionId: region.id, url: region.url, chosenAt: Date.now() });
    };
    socket.onerror = () => { clearTimeout(timer); reject(new Error('failed')); };
  }));

  // Promise.any resolves with the FIRST success and ignores failures, so an
  // unreachable region cannot delay or break the selection.
  return Promise.any(attempts);
}

export async function endpointFor(isReconnect: boolean): Promise<string> {
  const pinned = readPin();

  // During a reconnect, prefer the region we were just on: session resume and
  // any per-region state only work if we land back where we were.
  if (pinned && (isReconnect || Date.now() - pinned.chosenAt < RECONNECT_PIN_MS)) return pinned.url;
  if (pinned) return pinned.url;

  try {
    const chosen = await probe();
    localStorage.setItem('ws.region', JSON.stringify(chosen));
    return chosen.url;
  } catch {
    return REGIONS[0].url;                   // every probe failed: fall back to DNS
  }
}

On the DNS side, keep the record TTL short — 30 to 60 seconds — so a regional evacuation propagates quickly. The usual argument against short TTLs is query cost, which is negligible here because a WebSocket resolves once and then holds the connection for hours.

resource "aws_route53_record" "ws_eu" {
  zone_id        = var.zone_id
  name           = "ws.example.com"
  type           = "A"
  set_identifier = "eu-west-1"
  ttl            = 60                       # short: evacuation must propagate fast

  latency_routing_policy { region = "eu-west-1" }
  health_check_id = aws_route53_health_check.ws_eu.id
  records         = [var.eu_ip]
}

The health check is what makes evacuation automatic, and it must test something meaningful. An HTTP 200 from a load balancer proves the load balancer is up; a check that completes a real WebSocket upgrade proves the thing users actually need.

Verification #

Check what users actually get, from where they actually are.

# What does a resolver in another country receive?
dig +short ws.example.com @1.1.1.1
dig +short ws.example.com @8.8.8.8

# Does the provider honour EDNS Client Subnet?
dig +subnet=203.0.113.0/24 ws.example.com

# Measure the real cost: connect time, separately from message round trip
time npx wscat -c wss://ap.ws.example.com/probe --no-color

Synthetic probes from each region — one connect and one round trip per minute — give you the trend, and the number worth alerting on is connect time rather than message latency. A routing regression shows up there first, because the handshake pays several round trips and a message pays one.

The distribution matters more than the average. If p50 connect time is fine and p95 is triple, a subset of users is being routed to a distant region — usually the ones behind a public resolver, which is exactly the failure DNS-based routing has.

Connect cost is where regional routing pays off, and it is worth separating from message latency because the handshake pays several round trips where a message pays one.

Operational checklist #

• Keep DNS TTLs at 30–60 seconds; a WebSocket resolves once, so the query cost is irrelevant.
• Verify your DNS provider honours EDNS Client Subnet, or accept that resolver location is what you are routing by.
• Probe from the client on first load and cache the choice for about a day.
• Use a dedicated lightweight probe endpoint that completes a real upgrade, not an HTTP health check.
• Pin reconnects to the previously chosen region so session resume keeps working.
• Make the regional health check complete an actual WebSocket upgrade, not just a load-balancer 200.
• Alert on p95 connect time per region, not the average — routing regressions hide in the tail.
• Rehearse an evacuation and measure the settle time end to end.

FAQ #

Why do some users still get the wrong region? #

Because DNS routes resolvers. A user in Melbourne on a public resolver whose nearest node is in Singapore gets whatever is closest to Singapore, and neither your provider nor the user can tell. EDNS Client Subnet helps where it is supported, and client-side probing sidesteps the problem entirely — which is the main argument for paying its one-round-trip cost.

Is anycast better for WebSockets? #

It routes faster and needs no TTL, but it introduces a risk HTTP does not have: if BGP re-converges mid-session, packets for an established TCP connection arrive at a different site and the connection dies. Providers who run anycast for TCP mitigate this with careful announcement policy, and re-convergence is rare — but “rare” across millions of connection-hours is a steady background of unexplained 1006 closes. Use it when your provider handles the stickiness for you.

Should I route by IP geolocation instead? #

GeoIP tells you roughly where an address is registered, which correlates loosely with network distance and not at all with routing quality. A VPN, a mobile carrier’s central egress, or a corporate proxy makes it actively wrong. It is a reasonable last-resort fallback when no probe result is available and DNS is not latency-aware, but it should never be the primary mechanism.

How do I move users out of a failing region? #

Fail its health check so DNS stops handing it out, then drain its connections deliberately with 1001 so clients reconnect promptly. The receiving regions must have headroom for the surge — the arithmetic and the jitter requirement are in multi-region and edge WebSocket delivery, and the drain mechanics are the same as in draining WebSocket connections during deploys.

Does a CDN help route WebSockets? #

Only if it terminates them at the edge. A CDN that passes the connection through to your origin has moved nothing — every byte still travels the full distance. An edge platform that terminates TLS and the upgrade close to the user cuts connection setup substantially, which is the largest single win available for distant users, but message latency is unchanged unless the room state also lives at the edge.

Related #

• Multi-Region & Edge WebSocket Delivery — the wider architecture this routing serves.
• Cross-Region WebSocket State Sync — what has to be true for a client to be served by any region.
• Draining WebSocket Connections During Deploys — the drain mechanics an evacuation reuses.
• Exponential Backoff with Jitter for WebSocket Reconnects — why an evacuation without jitter is a stampede.
• Load Balancer Sticky Sessions — affinity within a region, once the region is chosen.

Back to Multi-Region & Edge WebSocket Delivery