Draining WebSocket Connections During Deploys #

The deploy is green in thirty seconds and the incident channel lights up anyway: every user saw “Reconnecting”, the new pods took a 40,000-connection spike in one second, half the handshakes timed out, and those clients retried into an already-saturated fleet. Nothing was down. The rollout simply severed every long-lived connection at once, which for a stateless HTTP service is a non-event and for a WebSocket fleet is an outage.

Draining is the difference, and Kubernetes gives you exactly the hooks needed — they just default to values written for request/response services.

Root cause #

When a pod is deleted, Kubernetes does two things concurrently: it sends SIGTERM to the container and it removes the pod from Service endpoints. Both are asynchronous, and neither knows anything about your connections.

Three defaults then work against a socket server:

terminationGracePeriodSeconds is 30. After that, SIGKILL. Thirty seconds is generous for finishing in-flight HTTP requests and far too short to drain sockets in a controlled, staggered way.

Endpoint removal is not instant. The pod’s removal must propagate to kube-proxy and to any ingress controller, which takes a second or two. A pod that exits immediately on SIGTERM therefore refuses connections that are still being routed to it — the classic “connection refused during deploy” that looks like a readiness bug.

Nothing tells the client this was deliberate. A process that exits without closing its sockets leaves every client observing 1006, indistinguishable from a network failure, so well-behaved clients back off exactly when you want them to reconnect promptly.

The last point is the one with the biggest payoff. A close frame with 1001 Going Away costs nothing and tells clients “come back, just not to me”, which turns a backoff-delayed reconnect into an immediate one.

Resolution #

The deployment sets a grace period long enough to drain and a preStop hook that buys time for endpoint propagation. The application does the staggered close itself.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ws-node
spec:
  replicas: 12
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1        # never drain two pods at once
      maxSurge: 2              # new capacity BEFORE old capacity leaves
  template:
    spec:
      # Long enough for the staggered drain plus headroom. SIGKILL at this point.
      terminationGracePeriodSeconds: 120
      containers:
        - name: ws
          lifecycle:
            preStop:
              exec:
                # Give kube-proxy and the ingress time to stop routing here
                # BEFORE the app starts closing sockets.
                command: ["/bin/sh", "-c", "sleep 10"]
          readinessProbe:
            httpGet: { path: /readyz, port: 8080 }
            periodSeconds: 2
            failureThreshold: 2
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: ws-node
spec:
  minAvailable: 10             # node drains and evictions cannot take the fleet below this
  selector:
    matchLabels: { app: ws-node }

import { WebSocketServer } from 'ws';

const DRAIN_WINDOW_MS = 60_000;      // spread closes across this window
const BATCH_INTERVAL_MS = 500;

let draining = false;

export function installDrain(wss: WebSocketServer, server: import('http').Server): void {
  // Readiness flips first: the probe fails, endpoints are removed, no new
  // connections arrive here — all before a single socket is closed.
  server.on('request', (req, res) => {
    if (req.url === '/readyz') { res.writeHead(draining ? 503 : 200).end(); }
  });

  process.on('SIGTERM', () => {
    draining = true;
    server.close();                                  // stop accepting upgrades

    const sockets = [...wss.clients];
    const batches = Math.max(1, Math.floor(DRAIN_WINDOW_MS / BATCH_INTERVAL_MS));
    const perBatch = Math.ceil(sockets.length / batches);
    let index = 0;

    // Staggered close: closing 40,000 sockets in one tick recreates exactly the
    // stampede this is meant to prevent, just from the server side.
    const timer = setInterval(() => {
      const batch = sockets.slice(index, index + perBatch);
      index += perBatch;
      // 1001 Going Away: the client should reconnect promptly rather than back off.
      for (const socket of batch) socket.close(1001, 'server draining');

      if (index >= sockets.length) {
        clearInterval(timer);
        // Short grace for close frames to flush, then exit before SIGKILL.
        setTimeout(() => process.exit(0), 5_000);
      }
    }, BATCH_INTERVAL_MS);
  });
}

The ordering is what makes this work. Readiness fails first so no new connections arrive; the preStop sleep covers the propagation delay; only then do sockets start closing, and they close in batches rather than all at once. A pod that skips any one of those three steps produces a visible disruption even though the other two are correct.

maxSurge: 2 with maxUnavailable: 1 is the other half: new pods are ready and receiving connections before old ones start draining, so the fleet never dips below full capacity while 40,000 clients are looking for somewhere to land.

Verification #

Roll a deploy and watch the three signals that matter: connection count, close-code mix, and handshake latency.

# Watch pods drain in order rather than together
kubectl get pods -l app=ws-node -w

# Confirm the drain actually staggers: connection count should ramp down, not cliff
kubectl exec deploy/ws-node -- curl -s localhost:9090/metrics | grep ws_connections_active

# The decisive check: clients should see 1001, not 1006
kubectl logs -l app=ws-node --tail=50 | grep -c 'server draining'

On the dashboard, a healthy rollout shows a 1001 spike and effectively no change in 1006. If 1006 rises during a deploy, sockets are being severed rather than closed — usually because the grace period expired mid-drain, or because the process exits on SIGTERM before the drain runs at all.

The second signal is handshake latency on the receiving pods. A drain that is too fast shows up as p99 upgrade time climbing into seconds while the surge lands; widen DRAIN_WINDOW_MS until it stays flat.

Operational checklist #

• Fail the readiness probe first, so endpoints are removed before any socket is closed.
• Add a preStop sleep of 5–15 seconds to cover endpoint propagation to kube-proxy and the ingress.
• Set terminationGracePeriodSeconds longer than your drain window plus a margin.
• Close in batches across a window rather than all at once; a server-side stampede is still a stampede.
• Always close with 1001 Going Away so clients reconnect promptly instead of backing off.
• Use maxSurge greater than zero and maxUnavailable: 1 so capacity arrives before it leaves.
• Add a PodDisruptionBudget so node drains and cluster autoscaling cannot evict several pods at once.
• Exit the process explicitly after the drain rather than waiting for SIGKILL.

FAQ #

Why does the preStop hook just sleep? #

Because the thing you are waiting for happens outside the pod. Removing the pod from Service endpoints has to propagate to every kube-proxy and to the ingress controller, and there is no signal delivered to the container when that finishes. A short sleep is the standard, if inelegant, way to cover the gap — without it, sockets you close are immediately replaced by new connections routed to the pod you are trying to empty.

How long should the drain window be? #

Long enough that the receiving pods absorb the surge without their handshake latency degrading. Start from the arithmetic: connections divided by window gives arrival rate, and that rate must sit comfortably inside what a pod can accept per second. For 40,000 connections across twelve pods, a sixty-second window produces roughly 55 new connections per second per pod, which is undemanding. Then verify against the real p99.

Do clients need changes for this to work? #

Only one, and it is worth making: distinguish 1001 from 1006 in the reconnect classifier. On 1001 reconnect immediately, because the server told you it was leaving deliberately; on 1006 back off, because the cause is unknown. That is the classification described in WebSocket close codes explained, and without it a perfectly drained deploy still leaves users waiting out a backoff window.

What about session state on the draining pod? #

It has to be recoverable elsewhere, or the drain simply moves the problem. Either keep per-connection state in shared storage, or give clients a resume token so the new pod can rebuild it — see resuming WebSocket sessions after reconnect. A deploy is the most frequent reason a client changes pods, so this is not an edge case.

Does this apply to autoscaling too? #

Yes, and it matters more there because scale-down is unattended. A HorizontalPodAutoscaler or KEDA scaler removing pods triggers the same termination path, so the same drain logic runs — but scale-down can happen at any hour with nobody watching. Set a conservative stabilizationWindowSeconds on scale-down so the fleet does not oscillate, as covered in autoscaling WebSockets on Kubernetes with KEDA.

Related #

• Horizontal Scaling on Kubernetes — the surrounding deployment model and capacity planning.
• Autoscaling WebSockets on Kubernetes with KEDA — scale-down triggers the same drain path unattended.
• WebSocket Close Codes Explained — why 1001 and 1006 produce very different client behaviour.
• Exponential Backoff with Jitter for WebSocket Reconnects — the client-side half of a survivable rollout.
• Resuming WebSocket Sessions After Reconnect — making a pod change invisible to the user.

Back to Horizontal Scaling on Kubernetes