Horizontal Scaling of WebSockets on Kubernetes #

A rolling deploy ships at 14:00. Kubernetes terminates the old ReplicaSet pod-by-pod, and every long-lived WebSocket pinned to those pods dies the instant the container receives SIGTERM. Thirty thousand browsers simultaneously fire onclose, trigger reconnect storms, and re-handshake against the new pods — a self-inflicted thundering herd on every deploy. Meanwhile your CPU-based HorizontalPodAutoscaler sees idle sockets, reads 8% CPU, and scales the Deployment down into the storm.

This is the core mismatch: Kubernetes primitives assume stateless, short-lived requests. WebSockets are stateful and long-lived. Running them on Kubernetes works well, but only once you stop autoscaling on CPU, drain connections on pod shutdown instead of killing them, and bound voluntary disruptions with a budget. This guide walks the Deployment manifest, the autoscaling signal, and the graceful-shutdown handler that make horizontal scaling safe.

Prerequisites #

Before scaling out, the connection layer below the pod must already be correct:

• A working upgrade path through the Ingress. The ingress controller (NGINX, Contour, or an ALB Ingress) must forward Upgrade/Connection headers and use a long enough idle timeout. See configuring Nginx for WebSocket upgrades.
• Externalized state. Any shared state must live in Redis or a database so a connection can land on any pod. If you still depend on in-memory affinity, read Load Balancer Sticky Sessions first — affinity and aggressive autoscaling fight each other during scale-down.
• Server-side lifecycle handling. Heartbeats, idle eviction, and clean close semantics belong to the app, covered under Backend WebSocket Connection Management.
• A metrics endpoint the pod already exposes (/metrics) reporting active connection count, since that number drives autoscaling.

Why CPU-Based HPA Is the Wrong Signal #

A WebSocket that has completed its handshake costs almost no CPU while idle — it is a file descriptor and a small buffer waiting on the kernel. A single pod can hold tens of thousands of mostly-silent connections at 5–10% CPU. CPU therefore tells you nothing about how full a pod is. Worse, it flaps: a burst of broadcast traffic spikes CPU, the HPA adds pods, the burst ends, CPU collapses, and the HPA scales back down — terminating pods that still hold thousands of live connections.

The correct signal is active connection count per pod. You scale up when pods approach their connection ceiling (file-descriptor limits, memory per connection) and you never scale down purely because traffic went quiet. The diagram below shows the data path and the autoscaling control loop driven by that metric.

Core implementation #

Two pieces work together: the Deployment manifest that tells Kubernetes how to start and stop the pod, and the application code that drains connections when SIGTERM arrives.

The Deployment manifest #

The critical knobs are terminationGracePeriodSeconds (how long Kubernetes waits before SIGKILL), a preStop hook that flips the pod out of the load-balancer rotation, and a readiness probe that returns unready the moment draining begins so the Service stops sending new connections.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ws-server
spec:
  replicas: 3
  selector:
    matchLabels: { app: ws-server }
  template:
    metadata:
      labels: { app: ws-server }
    spec:
      # Must exceed (preStop sleep + drain timeout) so the app finishes draining
      # before Kubernetes sends SIGKILL. 120s here = 5s sleep + ~110s drain budget.
      terminationGracePeriodSeconds: 120
      containers:
        - name: ws-server
          image: registry.example.com/ws-server:1.8.0
          ports:
            - containerPort: 8080
          lifecycle:
            preStop:
              exec:
                # Sleep so endpoint removal propagates to the Ingress/Service
                # BEFORE the app starts refusing new connections. Avoids a race
                # where kube-proxy still routes new clients to a draining pod.
                command: ["/bin/sh", "-c", "sleep 5"]
          readinessProbe:
            # Returns 503 once SIGTERM flips the draining flag, pulling the pod
            # out of Service endpoints so no NEW connections arrive.
            httpGet: { path: /readyz, port: 8080 }
            periodSeconds: 2
            failureThreshold: 1
          livenessProbe:
            # Stays healthy during drain — do NOT let liveness kill a draining pod.
            httpGet: { path: /healthz, port: 8080 }
            periodSeconds: 10
            failureThreshold: 3

The SIGTERM graceful-drain handler #

When the container gets SIGTERM, the app must (1) mark itself unready, (2) stop accepting new upgrades, (3) ask existing clients to reconnect with a close frame, and (4) wait for them to drain before exiting.

import { WebSocketServer } from "ws";
import http from "node:http";

const DRAIN_TIMEOUT_MS = 110_000; // must fit inside terminationGracePeriodSeconds
const GOING_AWAY = 1001;          // RFC 6455: "server is going away"

let draining = false;
const server = http.createServer((req, res) => {
  // Readiness probe: report unready the instant draining begins so the
  // Service removes this pod from endpoints and routes new clients elsewhere.
  if (req.url === "/readyz") {
    res.writeHead(draining ? 503 : 200).end();
    return;
  }
  if (req.url === "/healthz") {
    res.writeHead(200).end(); // liveness stays green throughout the drain
    return;
  }
  res.writeHead(404).end();
});

const wss = new WebSocketServer({ server });

function gracefulDrain(): void {
  if (draining) return;
  draining = true; // /readyz now returns 503 — no new connections from the Service

  // Tell every live client we are going away with a clean close frame so the
  // client's reconnect logic (backoff + jitter) kicks in against a healthy pod.
  for (const ws of wss.clients) {
    ws.close(GOING_AWAY, "server draining");
  }

  // Poll until all sockets have closed, then exit cleanly.
  const deadline = Date.now() + DRAIN_TIMEOUT_MS;
  const tick = setInterval(() => {
    if (wss.clients.size === 0 || Date.now() > deadline) {
      clearInterval(tick);
      // Force-close any stragglers that ignored the close frame, then exit.
      for (const ws of wss.clients) ws.terminate();
      server.close(() => process.exit(0));
    }
  }, 1_000);
}

// SIGTERM is what Kubernetes sends on pod termination AFTER the preStop hook.
process.on("SIGTERM", gracefulDrain);

server.listen(8080);

The sequence on the wire is: preStop sleep 5 runs first (endpoint removal propagates), then Kubernetes sends SIGTERM, gracefulDrain flips readiness and closes sockets, clients reconnect against pods that are still in rotation, and the process exits well before the 120s grace period elapses.

Configuration reference #

Edge cases & gotchas #

• The SIGTERM/endpoint-removal race. SIGTERM and endpoint removal are dispatched concurrently. Without the preStop sleep, the app can start refusing connections while kube-proxy on some nodes still routes new clients to it — they get a connection reset. The sleep buys time for endpoint propagation; it is not optional.
• Ingress idle timeout silently kills sockets. Many controllers default to a 60s proxy read timeout. A WebSocket with a 30s heartbeat survives; one with a 90s heartbeat gets disconnected mid-connection with no app-level error. Set the timeout to at least twice your heartbeat interval, mirroring the upgrade-header rules in configuring Nginx for WebSocket upgrades.
• Sticky routing during scale-down. If the Ingress uses session affinity, scaling down can repeatedly hash reconnecting clients back to terminating pods. Either disable affinity for WebSocket Services that externalize state, or make affinity respect readiness — see Load Balancer Sticky Sessions.
• Liveness probes that kill draining pods. If your liveness probe shares the readiness handler and you return 503 while draining, the kubelet will SIGKILL the pod mid-drain. Keep /healthz and /readyz separate, as shown above.

Verification #

Confirm a rolling deploy drains instead of mass-disconnecting:

# Watch the rollout; pods should leave gradually, not all at once.
kubectl rollout status deployment/ws-server --watch

# Exec into a terminating pod and count live sockets shrinking during drain.
kubectl exec -it ws-server-old-xxxx -- ss -tnp 'sport = :8080' | wc -l

# Confirm the readiness flip pulled the pod from Service endpoints.
kubectl get endpointslices -l kubernetes.io/service-name=ws-server -o wide

# Verify the PodDisruptionBudget is allowing the right number of disruptions.
kubectl get pdb ws-server -o jsonpath='{.status.disruptionsAllowed}'

During the drain window, ss on the terminating pod should show the connection count falling toward zero while the new pods’ counts rise — clients are reconnecting, not erroring. If ss stays flat at SIGTERM, your handler never received the signal (check that the container’s PID 1 forwards signals).

Guides in this area #

Autoscaling WebSockets on Kubernetes with KEDA walks the full ScaledObject definition that turns active-connection-count into a scaling decision, including the Prometheus trigger and stabilization windows that stop the HPA from flapping.

Deploys are the hard part #

A WebSocket fleet spends most of its life in steady state and most of its incidents in transitions, and the transition you perform most often is a deploy. For a stateless service that is a non-event; for a fleet holding tens of thousands of long-lived sockets it is the single most disruptive routine operation you run.

The default behaviour is unhelpful in three specific ways. terminationGracePeriodSeconds is 30, which is generous for finishing HTTP requests and far too short to drain sockets gradually. Endpoint removal is asynchronous, so a pod that exits promptly on SIGTERM refuses connections that are still being routed to it. And a process that exits without closing its sockets leaves every client observing 1006 — indistinguishable from a network fault — so well-behaved clients back off precisely when you want them to reconnect promptly.

Four changes make a rollout invisible. Fail readiness first, so endpoints are removed and no new connections arrive before anything is closed. Wait for propagation with a short preStop sleep, because there is no signal delivered to the container when kube-proxy and the ingress have caught up. Close deliberately with 1001 Going Away, which tells clients this was intentional and they should return immediately rather than back off. And stagger the closes across a window of roughly a minute, because closing 40,000 sockets in one tick recreates the stampede from the server side.

Surge matters too. maxSurge greater than zero with maxUnavailable: 1 brings new capacity online before old capacity leaves, so the fleet never dips below full while tens of thousands of clients are looking for somewhere to land. A rollout that drains before it surges is a rollout that runs at reduced capacity exactly when demand spikes.

The same machinery covers unplanned failures, which is the real reason to build it. A node that dies without draining produces the concentrated spike above, and a fleet running above roughly 70% utilisation has nowhere to put those connections. Jittered client reconnects and genuine headroom are what turn a lost pod into a blip rather than a cascade.

Scaling decisions need the same care in the opposite direction. Scale-down destroys live connections, so a fleet oscillating around a threshold gives users a reconnect every few minutes for no visible reason — which is why the stabilisation window on the way down should be minutes and on the way up should be near zero.

FAQ #

Why not just autoscale WebSocket pods on CPU? #

An idle WebSocket consumes negligible CPU, so CPU never reflects how many connections a pod holds. You can be at 95% of your file-descriptor ceiling while sitting at 8% CPU. Scale on active connection count per pod instead, and gate scale-down with a long stabilization window so a quiet period doesn’t terminate full pods.

How do I stop a rolling deploy from disconnecting every client? #

Combine a preStop sleep (so the pod leaves the load balancer before it refuses traffic), a SIGTERM handler that sends RFC 6455 close code 1001 and waits for sockets to drain, and a terminationGracePeriodSeconds larger than your drain timeout. Clients then reconnect against healthy pods using normal backoff instead of all hitting onclose simultaneously.

What does the preStop sleep actually accomplish? #

Kubernetes sends SIGTERM and removes the pod from Service endpoints at the same time. The sleep delays the app’s own shutdown long enough for endpoint removal to propagate through kube-proxy and the Ingress, closing the window where new clients could still be routed to a pod that is about to terminate.

Does this work behind an AWS ALB or NGINX Ingress? #

Yes, with two caveats: the controller must forward Upgrade/Connection headers, and its idle timeout must exceed twice your heartbeat interval or it will silently drop live sockets. ALB target deregistration delay plays the same role as the preStop sleep, so tune both together.

What does a PodDisruptionBudget protect against here? #

A PDB caps voluntary disruptions — node drains, cluster autoscaler scale-downs, kubectl drain. Setting minAvailable: 80% means a single node drain can’t evict every WebSocket pod simultaneously, so connections always have healthy pods to reconnect to. It does not protect against involuntary disruptions like node crashes.

What connection count should trigger scaling? #

Connections per pod, not CPU. A fleet holding 40,000 idle sockets can sit at 5% CPU, so CPU-based autoscaling will happily remove pods that are entirely full. Scale on the resource that is actually scarce — connections against your measured per-pod ceiling — and pair it with messages per second, which captures fan-out amplification that connection count alone misses.

How long should the termination grace period be? #

Longer than your drain window plus a margin: if you stagger closes across sixty seconds, 120 seconds is a reasonable grace period. The cost of a longer grace period is only that a genuinely wedged pod takes longer to disappear, which is far cheaper than the cost of SIGKILL arriving mid-drain and severing whatever remains.

Do I need a PodDisruptionBudget? #

Yes, and it is the setting most often missing. Without one, a node drain or an autoscaler scale-down can evict several pods simultaneously, producing exactly the concentrated reconnect spike the rollout strategy was designed to prevent. A minAvailable that keeps the fleet above its real capacity floor makes those operations safe by construction.

Related #

• Autoscaling WebSockets on Kubernetes with KEDA — the full KEDA ScaledObject and Prometheus trigger for connection-count scaling.
• Load Balancer Sticky Sessions — session affinity trade-offs that interact with scale-down.
• Configuring Nginx for WebSocket Upgrades — upgrade headers and idle timeouts the Ingress must honor.
• Backend WebSocket Connection Management — application-level lifecycle, heartbeats, and clean close semantics.

Back to Scaling Real-Time Infrastructure