Rebalancing WebSocket connections after scale-out #

The autoscaler did its job: CPU on your six WebSocket pods crossed the threshold and four new pods came up. Ten minutes later the six old pods are still at 90% CPU and the four new ones are nearly idle. Load balancers only balance new connections, and WebSocket clients do not make new connections — they stay on whatever pod they reached hours ago. Scale-out therefore helps only as fast as clients churn naturally, which for a long-lived dashboard can be never. The same skew appears after rolling restarts, where the last pod to restart ends up nearly empty. Fixing it requires the fleet to actively shed connections from overloaded pods, gently enough that the reconnects do not become a problem of their own.

Root cause #

Load balancing decisions happen at connection time. Round robin, least connections and every other algorithm choose a backend when the TCP connection or the upgrade request arrives, and never revisit that choice. For HTTP traffic that is fine, because each request is a new decision point. A WebSocket is one decision that lasts for the life of the connection.

After a scale-out, the new pods receive only the connections that are opened from then on: new users, and existing users who happen to reconnect. With a typical daily churn, it can take hours for the distribution to even out; with long-lived sessions, the old pods may stay overloaded until the next deploy. Rolling restarts produce the mirror image: each restarted pod’s clients reconnect across the remaining pods, so pods restarted early accumulate connections and the last pod restarted starts almost empty — the skew described in draining WebSocket connections during deploys.

Without rebalancing, the distribution after a restart reflects the order pods restarted in, not their capacity.

Resolution #

Give every pod a view of the fleet’s average load and a gentle shedding loop: when a pod holds meaningfully more than its share, it closes a small fraction of its connections per interval with close code 1012 Service Restart or 1013 Try Again Later, letting clients reconnect through the load balancer, which routes them — least-connections or readiness permitting — to the emptier pods. Shed slowly, stop inside a tolerance band, and never shed during a fleet-wide event.

import type { WebSocketServer, WebSocket } from 'ws';
import { createClient } from 'redis';

const REPORT_INTERVAL_MS = 10_000;
const SHED_INTERVAL_MS = 5_000;
const TOLERANCE = 1.15;               // tolerate up to 15% above the fleet average
const MAX_SHED_FRACTION = 0.01;       // at most 1% of this pod's connections per interval
const MIN_FLEET_PODS = 2;
const CLOSE_REBALANCE = 1013;         // "try again later": client reconnects with jitter

const redis = createClient(); await redis.connect();
const POD = process.env.HOSTNAME!;

export function startRebalancer(wss: WebSocketServer) {
  // Each pod reports its count with a short TTL; dead pods drop out automatically.
  setInterval(async () => {
    await redis.set(`conns:${POD}`, String(wss.clients.size), { EX: 30 });
  }, REPORT_INTERVAL_MS).unref();

  setInterval(async () => {
    const keys = await redis.keys('conns:*');           // small fleet; use a set/hash at scale
    if (keys.length < MIN_FLEET_PODS) return;
    const counts = (await redis.mGet(keys)).map(Number);
    const avg = counts.reduce((a, b) => a + b, 0) / counts.length;
    const mine = wss.clients.size;
    if (mine <= avg * TOLERANCE) return;                // inside the band: do nothing

    // Shed only the excess, capped per interval so reconnects stay a trickle.
    const excess = mine - Math.floor(avg);
    const quota = Math.min(excess, Math.max(1, Math.floor(mine * MAX_SHED_FRACTION)));
    let shed = 0;
    for (const ws of wss.clients) {
      if (shed >= quota) break;
      if (!isSheddable(ws)) continue;                   // skip clients mid-upload, etc.
      ws.close(CLOSE_REBALANCE, 'rebalancing');
      shed += 1;
    }
    metrics.rebalanceShed.inc(shed);
  }, SHED_INTERVAL_MS).unref();
}

// Prefer idle or recently reconnected clients; avoid interrupting active work.
function isSheddable(ws: WebSocket & { lastActivityAt?: number; busy?: boolean }) {
  return !ws.busy && Date.now() - (ws.lastActivityAt ?? 0) > 30_000;
}
declare const metrics: { rebalanceShed: { inc(n: number): void } };

The load balancer must actually send reconnecting clients to the emptier pods. With plain round robin, a shed client lands on a random pod — possibly the one it just left — so rebalancing converges slowly. Least-connections balancing (HAProxy balance leastconn, nginx least_conn, Envoy LEAST_REQUEST) makes each shed client move to an emptier pod. Consistent-hash affinity works against rebalancing; if you use it, rebalancing must move clients by changing the hash ring (adding the new pods) rather than by shedding. Readiness gating from WebSocket readiness and liveness probes complements shedding: pods over capacity stop accepting new clients, so reconnects cannot land back on them.

Clients must handle 1013 well: reconnect with jitter, resume from their last sequence number, and not surface an error to the user. The resume mechanics are in resuming WebSocket sessions after reconnect.

Rebalance in minutes, not in one burst.

Edge cases #

Fleet-wide events. During a deploy, a regional failover or an incident, every pod’s count changes at once and shedding would add churn to churn. Suspend rebalancing while a deploy is in progress or while the fleet’s total connections are changing quickly.

Heterogeneous pods. If pods have different sizes, compare each pod’s count to its capacity share rather than a plain average, or pods on bigger nodes will be pushed down to the level of smaller ones.

Sticky state. Pods that hold significant per-connection state in memory (subscriptions, resume buffers) make moves more expensive. Keep that state in a shared store so a moved client resumes cheaply, or shed more slowly.

Verification #

Scale a staging deployment from six to ten replicas under a steady load of idle connections and watch per-pod connection counts converge:

kubectl scale deploy/realtime --replicas=10
# Per-pod connections every 30 s (from each pod's metrics endpoint).
watch -n 30 'for p in $(kubectl get pods -l app=realtime -o name); do
  echo "$p $(kubectl exec $p -- curl -s localhost:9464/metrics | grep ^ws_connections_open | cut -d" " -f2)"; done'

Pass criteria: every pod within the tolerance band within your target time, the reconnect rate during rebalancing a small fraction of the fleet’s handshake capacity, and no user-visible errors — clients’ resume should make each move invisible. Graph the coefficient of variation of connections per pod as an ongoing health metric; it should stay low except briefly after deploys.

Gradual shedding turns a scale-out into an actual capacity increase.

Operational checklist #

Pods publish their connection counts to a shared store with a TTL.
Each pod sheds only its excess above a tolerance band, capped per interval.
Shed connections close with 1013 (or 1012) and clients reconnect with jitter and resume.
The load balancer uses least-connections so reconnects land on emptier pods.
Rebalancing pauses during deploys and fleet-wide incidents.
Idle clients are preferred; clients mid-operation are skipped.
The spread of connections per pod is graphed and alerted on.

FAQ #

Why don’t new pods get WebSocket traffic after scaling out? #

Load balancers only place new connections, and existing WebSocket clients stay connected indefinitely. New pods receive only newly opened connections until the fleet actively moves some clients.

Won’t closing connections hurt users? #

Not if it is gradual and clients resume cleanly: a reconnect with session resume takes well under a second and can be invisible. Closing a small percentage per interval avoids a reconnect storm.

Which close code should rebalancing use? #

1013 Try Again Later or 1012 Service Restart — both signal a temporary server-side condition that clients should retry after a short delay. Avoid 1008, which many clients treat as permanent.

Can I rebalance with consistent hashing? #

Consistent hashing keeps clients on the pod their key maps to, so shedding them just sends them back. Rebalancing happens by changing the ring — adding the new pods moves roughly 1/N of keys — and clients on remapped keys must be closed so they reconnect to their new owner.

Draining WebSocket Connections During Deploys — the source of post-deploy skew.
Autoscaling WebSockets on Kubernetes with KEDA — the scale-out that needs rebalancing.
WebSocket Sticky Sessions with nginx ip_hash — balancing methods and their effect on moves.
Rate Limiting WebSocket Handshakes — keeping rebalance reconnects under the accept budget.

Back to Horizontal Scaling on Kubernetes.