WebSocket readiness and liveness probes #

During a traffic spike, your WebSocket pods start failing their liveness probes — the event loop is busy fanning out messages and the /healthz handler answers late — and Kubernetes restarts them. Each restart drops forty thousand connections onto the remaining pods, which get busier, fail their probes, and restart too. A health check meant to improve availability has turned a load spike into a cascading outage. The opposite failure is just as common: a pod whose Redis connection died keeps passing its probes, stays in the Service, and accepts new WebSocket clients that will never receive a broadcast. Probes for long-lived connection servers need a different design from probes for stateless HTTP services, because restarting a pod is far more expensive and “healthy” means more than “the process answers”.

Root cause #

Kubernetes has three probes with distinct consequences. A failing liveness probe restarts the container. A failing readiness probe removes the pod from Service endpoints, so no new traffic is routed to it — existing connections are untouched. A startup probe holds off the other two until the application has initialised.

For stateless services the difference barely matters: restarting a pod loses a few in-flight requests. For a WebSocket server, a restart severs every connection on the pod and triggers a reconnect storm across the fleet. That makes liveness the most dangerous probe to get wrong. Two mistakes dominate. Liveness checks that measure load — a handler that times out when the event loop is busy, or a check that includes dependencies like Redis — restart pods for being busy or for another system’s outage, both of which a restart cannot fix. Readiness checks that measure nothing — returning 200 whenever the process is up — keep routing new clients to pods that cannot serve them properly: dependencies are down, the pod is draining for a deploy, or it is already full.

Liveness asks 'is it stuck?'; readiness asks 'should it take new clients?'

Resolution #

Expose two endpoints with different logic. Liveness answers from a timer-driven heartbeat: a setInterval records when the event loop last ran, and the probe fails only if that timestamp is very stale — a genuine deadlock or infinite loop, not ordinary load. Readiness combines dependency health, capacity, event-loop lag and a draining flag. Serve both from a tiny HTTP server so they are not queued behind WebSocket traffic.

import http from 'node:http';
import { monitorEventLoopDelay } from 'node:perf_hooks';
import type { WebSocketServer } from 'ws';

const LOOP_TICK_MS = 1_000;
const LIVENESS_STALE_MS = 30_000;          // only a truly stuck loop fails liveness
const MAX_CONNECTIONS = 40_000;            // from the capacity plan for this pod size
const READY_HEADROOM = 0.95;               // stop taking new clients at 95% of capacity
const MAX_LOOP_LAG_P99_MS = 250;           // saturation signal for readiness

let lastTick = Date.now();
setInterval(() => { lastTick = Date.now(); }, LOOP_TICK_MS).unref();

const loopDelay = monitorEventLoopDelay({ resolution: 20 });
loopDelay.enable();

export const state = { draining: false, pubsubHealthy: false };

export function startProbeServer(wss: WebSocketServer, port = 9102) {
  http.createServer((req, res) => {
    if (req.url === '/livez') {
      // Liveness: is the event loop running at all? Never checks dependencies or load.
      const stale = Date.now() - lastTick;
      res.writeHead(stale < LIVENESS_STALE_MS ? 200 : 500).end(`tick_age_ms=${stale}`);
      return;
    }
    if (req.url === '/readyz') {
      const conns = wss.clients.size;
      const lagP99 = loopDelay.percentile(99) / 1e6;       // ns → ms
      loopDelay.reset();
      const reasons: string[] = [];
      if (state.draining) reasons.push('draining');
      if (!state.pubsubHealthy) reasons.push('pubsub_down');
      if (conns >= MAX_CONNECTIONS * READY_HEADROOM) reasons.push('at_capacity');
      if (lagP99 > MAX_LOOP_LAG_P99_MS) reasons.push('loop_saturated');
      res.writeHead(reasons.length ? 503 : 200).end(JSON.stringify({ conns, lagP99, reasons }));
      return;
    }
    res.writeHead(404).end();
  }).listen(port);
}

# Pod spec excerpt
containers:
  - name: realtime
    ports: [{ name: ws, containerPort: 8080 }, { name: probes, containerPort: 9102 }]
    startupProbe:                      # allow slow warm-up (loading state, connecting to Redis)
      httpGet: { path: /livez, port: probes }
      periodSeconds: 2
      failureThreshold: 30
    livenessProbe:                     # generous: restart only a genuinely stuck process
      httpGet: { path: /livez, port: probes }
      periodSeconds: 10
      timeoutSeconds: 5
      failureThreshold: 6              # ~60 s of consecutive failure before restart
    readinessProbe:                    # responsive: stop routing new clients quickly
      httpGet: { path: /readyz, port: probes }
      periodSeconds: 5
      timeoutSeconds: 2
      failureThreshold: 2
      successThreshold: 1

Readiness failing does not disconnect existing clients; it only stops new ones arriving. That is precisely the right behaviour for every readiness reason above. When the pod is draining, pair not-ready with a graceful close sequence, as in draining WebSocket connections during deploys. When pub/sub is down, existing clients are not receiving broadcasts either; decide whether to close them with 1013 so they reconnect to healthy pods, or keep them while the dependency recovers.

Do not use a WebSocket connection as the probe itself. Opening a socket every few seconds from every kubelet consumes the resources the probe is supposed to measure and fails for reasons unrelated to health, such as authentication.

Dependency failures belong to readiness, never to liveness.

Edge cases #

Probes during a GC pause or burst. A long garbage-collection pause can delay the probe server too. Liveness tolerates it through a high failure threshold; readiness may briefly fail, which is harmless — the pod stops accepting new clients for a few seconds.

Capacity-based readiness and autoscaling. A pod that reports not-ready at capacity sheds new connections to other pods, which is what you want — but if every pod is at capacity, the Service has no endpoints and new clients get errors. Make sure the autoscaler scales on connection count before pods reach the readiness headroom; see autoscaling WebSockets on Kubernetes with KEDA.

Ingress health checks. Cloud load balancers in front of the ingress run their own health checks against nodes or pods. Point them at the readiness endpoint, not the WebSocket path, and align their thresholds so the two views of health agree.

Verification #

Test each readiness reason and confirm liveness never flips. Block the pod’s Redis traffic with a network policy and watch the endpoint list; the pod should leave it within one or two probe periods and existing connections should stay open:

kubectl get endpointslices -l kubernetes.io/service-name=realtime -o wide -w &
kubectl exec deploy/realtime -- curl -s localhost:9102/readyz          # {"reasons":["pubsub_down"],...}
kubectl get pods -l app=realtime -o custom-columns=NAME:.metadata.name,RESTARTS:.status.containerStatuses[0].restartCount

Then load-test to saturation: event-loop lag rises, readiness fails, new connections go elsewhere — and the restart count stays at zero. Any restart during a load test means the liveness probe is measuring load.

Split probes restart only what a restart can fix.

Operational checklist #

Liveness reflects only event-loop progress, with a generous timeout and failure threshold.
Readiness reflects dependencies, capacity headroom, event-loop saturation and draining.
Probes are served on a separate lightweight port, not the WebSocket path.
A startup probe covers warm-up so liveness does not kill slow starts.
Draining sets readiness false before closing connections.
Autoscaling triggers before pods reach readiness capacity headroom.
Load tests confirm zero restarts under saturation.

FAQ #

Should the liveness probe check Redis or the database? #

No. If a dependency is down, restarting your pod does not fix it and drops every WebSocket connection on the pod. Put dependency checks in readiness, which stops new traffic without disconnecting existing clients.

Does a failing readiness probe disconnect WebSocket clients? #

No. Readiness only controls whether the pod receives new connections through the Service. Existing connections stay open until your code closes them.

Can I use a WebSocket handshake as a health check? #

It is possible but a poor idea: it consumes connection capacity, depends on authentication and routing, and makes probes fail for reasons unrelated to health. A plain HTTP endpoint on a separate port is simpler and more accurate.

What timeouts should liveness probes use for WebSocket servers? #

Generous ones: a few seconds of timeout and enough failures to span about a minute before a restart. The cost of a false restart — a reconnect storm — is far higher than the cost of noticing a real deadlock a little later.

Draining WebSocket Connections During Deploys — readiness as the first step of shutdown.
Autoscaling WebSockets on Kubernetes with KEDA — scaling before capacity is reached.
Rebalancing WebSocket Connections After Scale-Out — what readiness cannot do on its own.
Alerting on WebSocket Connection Drops — catching restart-driven drops.

Back to Horizontal Scaling on Kubernetes.