Autoscaling WebSockets on Kubernetes with KEDA #

Your WebSocket pods are pinned at 60,000 sockets each, but the HorizontalPodAutoscaler refuses to add replicas because average CPU sits at 8%. Idle-but-connected sockets consume file descriptors, heap, and the per-connection memory of your framework — none of which shows up on a CPU graph. By the time CPU finally spikes, the fleet is already past its connection ceiling and new clients get ECONNREFUSED or land on a pod that OOMKills mid-handshake. The fix is to stop scaling on CPU and start scaling on the one metric that actually tracks load: active connection count.

Root cause #

The default HorizontalPodAutoscaler scales on resource metrics — CPU and memory utilization averaged across the pods in a Deployment. That model assumes work is proportional to CPU. For request/response HTTP it usually is. For long-lived WebSockets it is not.

A WebSocket connection that has completed its upgrade and is sitting idle does almost no CPU work. It costs:

  • One file descriptor (and a slot against the pod’s nofile ulimit).
  • A kernel socket buffer pair (send/receive), typically tens of KB each.
  • Per-connection heap in your runtime — buffers, the ws socket object, any subscription/presence state you attach.

A Node.js pod can hold tens of thousands of these before memory pressure appears, and CPU stays near idle the entire time because no frames are flowing. So the HPA’s CPU signal is flat right up until the moment a pod runs out of descriptors or heap, at which point accept() starts failing and the process degrades non-gracefully. Memory-based HPA is only marginally better: heap fragmentation and GC make the utilization number noisy and lagging, and you still cannot answer “are we near the connection ceiling?” directly.

The metric that is proportional to load is the number of active connections per pod. If each pod is sized for, say, 8,000 connections, you want Kubernetes to add a replica whenever the fleet-wide average crosses a target like 6,000. CPU and memory cannot express that. You need an external metric, and that is exactly what KEDA provides — it acts as a metrics adapter that feeds a custom metric (here, scraped from Prometheus) into the same HPA machinery, so you can scale on ws_connections_active instead of CPU.

This is also why connection-count scaling pairs with load balancer sticky sessions: the autoscaler changes the number of pods, but the load balancer still has to pin each long-lived socket to one pod, and respect that pinning across scale events.

KEDA autoscaling flow on active connections WebSocket pods export ws_connections_active to Prometheus; KEDA reads it and drives the HPA to scale the Deployment. WS Pods /metrics gauge Prometheus scrapes gauge KEDA ScaledObject HPA Deployment replicas Scale signal = avg active connections per pod CPU stays flat while sockets fill the ceiling

Resolution #

First, the application must export the metric. KEDA scales nothing without a real number to read, so expose a gauge that increments on connection and decrements on close. With prom-client in a Node.js service running ws:

import { WebSocketServer } from 'ws';
import { Gauge, register } from 'prom-client';
import http from 'node:http';

// A gauge, not a counter: it must go DOWN when a socket closes.
const activeConnections = new Gauge({
  name: 'ws_connections_active',
  help: 'Currently open WebSocket connections on this pod',
});

const wss = new WebSocketServer({ port: 8080 });

wss.on('connection', (socket) => {
  activeConnections.inc();              // one more open socket
  socket.on('close', () => {
    activeConnections.dec();            // decrement so KEDA sees drains
  });
});

// Prometheus scrape endpoint on a separate port from the WS traffic.
http.createServer(async (_req, res) => {
  res.setHeader('Content-Type', register.contentType);
  res.end(await register.metrics());   // exposes ws_connections_active
}).listen(9090);

Then point KEDA at the Prometheus query. The ScaledObject below targets an average of 6,000 connections per pod. KEDA computes desired replicas as ceil(sum(ws_connections_active) / threshold), so a fleet holding 30,000 sockets settles on 5 pods.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: ws-gateway-scaler
  namespace: realtime
spec:
  scaleTargetRef:
    name: ws-gateway          # the Deployment to scale
  minReplicaCount: 3          # never below 3 for HA + spread
  maxReplicaCount: 40         # cap so a metric bug can't fan out infinitely
  cooldownPeriod: 300         # wait 5m of no activity before scaling to min
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          # Long window: do NOT yank pods that still hold live sockets.
          stabilizationWindowSeconds: 600
          policies:
            - type: Pods
              value: 1        # remove at most 1 pod per period
              periodSeconds: 120
        scaleUp:
          stabilizationWindowSeconds: 0   # add capacity immediately
          policies:
            - type: Pods
              value: 4
              periodSeconds: 60
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring.svc:9090
        metricName: ws_connections_active
        # Average target per replica; KEDA divides the sum by this.
        threshold: "6000"
        query: sum(ws_connections_active{app="ws-gateway"})

The dangerous direction is scale-down. When the HPA decides to remove a pod, Kubernetes sends SIGTERM and the pod still holds live sockets that will all break at once. A long scaleDown.stabilizationWindowSeconds (here 600s) stops the autoscaler from thrashing pods in and out on brief dips, but it does not drain them — you also need a preStop hook and a terminationGracePeriodSeconds long enough for clients to migrate. In the pod spec, send a WebSocket close frame to every connection, stop accepting new ones, and sleep until the gauge hits zero or the grace period expires. Pair that drain with load balancer sticky sessions so reconnecting clients land on a surviving pod rather than the one being torn down. This connection-count approach is the Kubernetes-specific piece of scaling real-time infrastructure; the broader horizontal-scaling concerns live in Horizontal Scaling on Kubernetes.

Operational checklist #

Autoscaling a connection fleet fails differently from autoscaling a request fleet, and the difference is entirely in the scale-down direction.

Why CPU is the wrong scaling signal here A comparison of scaling signals showing that CPU does not track idle connection load at all, while connections per pod and message throughput do. Why CPU is the wrong scaling signal here Tracks load? Scale-down risk CPU utilisation no — idle sockets cost none drops live connections Memory partly lags the real signal Active connections yes needs a drain Connections per pod yes best signal Messages per second yes, for fan-out pair with connections A fleet holding 40,000 idle sockets can sit at 5% CPU — scaling on CPU would remove pods that are entirely full
Scale on the resource that is actually scarce. For a socket fleet that is connections, not cycles.
Reconnect arrivals after a node drops Reconnect attempts per second for 25000 dropped clients: a fixed retry delay lands every client in one second, full jitter spreads them across 45 seconds. Reconnect arrivals after a node drops 25k clients, 45s window — fixed delay vs full jitter Fixed delay Full jitter 0 10k 20k 30k 25k 1s 3s 5s 7s 9s 11s 13s 15s 17s 19s 21s 23s 25s 27s 29s 31s 33s 35s 37s 39s 41s 43s 45s
What a scale-down event looks like from the clients' side: without a staggered drain and jittered reconnects, every socket on the removed pod comes back inside a second.

FAQ #

Why not just use CPU-based HPA for WebSocket pods? #

Because idle-but-connected sockets burn file descriptors and heap while doing almost no CPU work. CPU utilization stays flat until a pod hits its connection ceiling and starts failing accept(), which is far too late to add capacity. Active-connection count is the only signal that rises in step with actual load.

Does KEDA replace the HorizontalPodAutoscaler? #

No. KEDA creates and manages an HPA under the hood and feeds it an external metric. You still get standard HPA behavior — including the scaleDown/scaleUp behavior block — but the scale signal comes from your Prometheus query instead of CPU or memory.

What happens to open connections when KEDA scales down a pod? #

Kubernetes sends SIGTERM and those sockets close. Use a long scaleDown.stabilizationWindowSeconds to avoid thrashing, a preStop hook that close-frames clients and drains, and a terminationGracePeriodSeconds long enough for the drain. Sticky sessions then route reconnects to surviving pods.

Can I scale on a metric other than raw connection count? #

Yes. Any Prometheus expression works — messages/sec per pod, send-buffer backlog, or a derived ws_connections_active / pod_connection_capacity ratio. Connection count is the cleanest starting point because it maps directly to the descriptor and memory limits that actually break a pod.

What changes for Socket.IO vs raw ws? #

The pattern is identical — increment a gauge on connection, decrement on disconnect. Socket.IO adds polling fallbacks and a heartbeat layer, so make sure you count only fully-upgraded WebSocket transports (or accept that long-poll sessions count too) and that your drain logic emits Socket.IO’s close, not just a raw frame.

Why is scale-down riskier than scale-up for WebSockets? #

Because removing a pod destroys live connections, whereas adding one only creates capacity. Every scale-down is therefore an unplanned reconnect event for everyone on the removed pod, and if it happens repeatedly — a fleet oscillating around a threshold — users experience a reconnect every few minutes for no visible reason. Set a long stabilisation window on scale-down and a short one on scale-up.

What stabilisation window should I use? #

Minutes rather than seconds on the way down — 300 seconds is a reasonable starting point — and near zero on the way up. Connections are long-lived, so a brief dip in count says almost nothing about whether capacity is genuinely surplus, and reacting to it costs real users a reconnect. Scaling up quickly costs only money.

Back to Horizontal Scaling on Kubernetes