Fixing Slow Consumer Memory Growth in ws #

RSS on two pods out of twelve climbs about 40 MB an hour and never comes back down. There is no obvious leak: connections are removed from the map on close, listeners are unregistered, and the same code runs fine on the other ten pods. Then you take a heap snapshot and the answer is right at the top of the retainers list — tens of thousands of Buffer objects, retained by an array inside Sender, retained by a WebSocket, retained by your client set.

This is not a leak in the usual sense. Every object has a legitimate owner and will be freed the moment the socket drains or closes. The problem is that it will never drain, and nothing in your code is watching.

Root cause #

A slow consumer is a client that acknowledges TCP segments more slowly than you produce them. When that happens, the receive window shrinks, the kernel refuses more data, and ws holds outgoing frames in a userspace queue. Three client behaviours produce it, and they look identical from the server:

Genuinely slow networks. A mobile client on a congested cell keeps reading, just slowly. The queue grows during bursts and drains between them. This client recovers on its own if you stop feeding it faster than it drinks.

Suspended clients. A laptop lid closes, a phone backgrounds the tab, an OS suspends the process. The socket is alive at the TCP level — the kernel may still ACK — but the application never reads. The queue grows monotonically and does not recover until the device wakes, which may be hours.

Black-holed connections. A NAT mapping expires or a middlebox silently drops the flow. No FIN, no RST, no error event. The queue freezes at whatever it was and stays there forever, holding its memory until the process restarts.

The third case is the one that defeats naive cleanup, because the usual remedy — socket.close() — initiates a closing handshake and waits for a reply that will never come. The memory is not released; you have simply added a timer to the problem.

Retainer path for a slow consumer's queued frames The client map retains a WebSocket, which retains its Sender, which retains an array of queued Buffers; the buffers are only freed when the socket drains or is terminated. What the heap snapshot is telling you clients: Set 20,000 entries WebSocket one slow client Sender queue array Buffer × 38,412 retained size 148 MB never drains close() — starts a handshake and waits; a black-holed peer never replies, so nothing is freed terminate() — destroys the socket immediately; the queue is dropped and collected on the next GC Rule: close() for clients that are talking, terminate() for clients that have gone silent A stalled queue plus a missed pong is conclusive evidence of the second case

Resolution #

Give every connection a memory budget and a stall detector, and run both on the heartbeat sweep you already have. The budget bounds the damage a slow client can do; the stall detector distinguishes “slow” from “gone” and picks the right kill switch.

import { WebSocketServer, WebSocket } from 'ws';

const SWEEP_INTERVAL_MS = 30_000;
const QUEUE_BUDGET_BYTES = 4 * 1024 * 1024;    // per-connection memory allowance
const STALL_SWEEPS_BEFORE_TERMINATE = 2;       // no drain across two sweeps = gone

interface Tracked {
  lastQueue: number;
  stalledSweeps: number;
  isAlive: boolean;
}

const state = new WeakMap<WebSocket, Tracked>();

export function track(wss: WebSocketServer): NodeJS.Timeout {
  wss.on('connection', (socket) => {
    state.set(socket, { lastQueue: 0, stalledSweeps: 0, isAlive: true });
    socket.on('pong', () => { const t = state.get(socket); if (t) t.isAlive = true; });
  });

  return setInterval(() => {
    for (const socket of wss.clients) {
      const t = state.get(socket);
      if (!t) continue;

      const queued = socket.bufferedAmount;

      // 1. Over budget: this one client is costing more memory than it is worth.
      if (queued > QUEUE_BUDGET_BYTES) {
        socket.close(1013, 'client too slow');   // polite: it is still reading something
      }

      // 2. Not draining at all between sweeps — the peer has stopped consuming.
      if (queued > 0 && queued >= t.lastQueue) t.stalledSweeps += 1;
      else t.stalledSweeps = 0;
      t.lastQueue = queued;

      // 3. Stalled AND missed its pong: nothing is coming back. Destroy it now,
      //    because close() would wait for a handshake that cannot complete.
      if (t.stalledSweeps >= STALL_SWEEPS_BEFORE_TERMINATE && !t.isAlive) {
        socket.terminate();
        continue;
      }

      t.isAlive = false;
      socket.ping();
    }
  }, SWEEP_INTERVAL_MS);
}

Using a WeakMap keyed on the socket means the tracking state cannot itself become the leak — when the socket is collected, so is its entry. That matters more than it sounds: several “WebSocket memory leak” reports turn out to be the bookkeeping added to diagnose the original problem.

Heap needed per node, by connection count Estimated heap for 5,000 to 100,000 concurrent sockets at 34 kilobytes per socket plus 8 kilobytes of application state, against a 1024 megabyte pod limit. Heap needed per node, by connection count 34 KB socket + 8 KB app state each — dashed line is a 1024 MB pod limit Socket + buffers App state 5.0k 205 MB 10k 410 MB 830 MB 25k 1.0k MB 1.7k MB 50k 2.1k MB 3.3k MB 781 MB 100k 4.1k MB 1024 MB pod limit
Baseline heap by connection count at 34 KB of socket plus 8 KB of app state each. A single slow consumer holding a 148 MB queue is worth roughly 3,500 healthy connections.

That comparison is the argument for a per-connection budget. On a pod sized for 50,000 sockets, twenty stalled clients with unbounded queues can consume more memory than the entire healthy fleet, and they will do it silently, over hours, on whichever pod happened to receive them.

Verification #

Confirm the diagnosis before changing behaviour, then confirm the fix.

# Capture a heap snapshot from a running pod and look at retained size by constructor
kill -USR2 "$(pgrep -f 'node .*server.js')"        # writes Heap.<pid>.heapsnapshot

# Live view: sockets with a non-empty send queue, worst first
node -e "require('./inspect').top(20)"             # sorts wss.clients by bufferedAmount

# OS-side confirmation that the peer is not reading (Send-Q stuck, non-zero)
ss -tni state established '( sport = :8080 )' | grep -A1 Send-Q | head -20

In the snapshot, filter by Buffer and sort by retained size; the retainer path should read Buffer → (array) → Sender → WebSocket. After deploying the sweep, the same pod should show a bounded number of tracked sockets, ws_terminated_stalled_total incrementing occasionally, and RSS that plateaus instead of climbing. If RSS still climbs with an empty queue everywhere, the memory is not backpressure at all — look at your own per-connection state next.

The three client behaviours need different responses, and telling them apart is the whole diagnostic skill here.

Telling the three slow-consumer cases apart A slow network drains between bursts and still returns pongs; a suspended client grows without draining but eventually responds; a black-holed connection freezes and never responds, so only terminate reclaims its memory. Telling the three slow-consumer cases apart Queue depth Pong Right action Slow network grows then drains arrives coalesce, wait Suspended client grows monotonically arrives late close 1013 Black-holed frozen, no change never arrives terminate() Queue depth alone is ambiguous — pair it with the heartbeat result before choosing close or terminate
The pong result is what disambiguates a queue that will drain from one that never will.

Operational checklist #

FAQ #

How do I tell a slow consumer from an application memory leak? #

Look at whether the memory is retained by socket internals or by your own structures. A slow consumer shows Buffer objects under Sender; an application leak shows your own objects — a Map of sessions that never deletes, an array of message history, closures capturing request context. The other giveaway is distribution: slow-consumer growth is concentrated on a few pods and correlates with a handful of connections, while an application leak grows evenly everywhere.

Should I just raise the pod memory limit? #

Only as a temporary measure while you deploy the fix. Unbounded is unbounded: a larger limit buys hours, not safety, and it makes the eventual OOM kill more disruptive because more connections are lost at once. Bound the per-connection cost and the limit stops being the thing standing between you and an outage.

Does terminate() lose messages? #

Yes — everything still queued is discarded, which is the point. Those messages were never going to arrive; keeping them costs memory and delays nothing but the inevitable. If some of them mattered, they belong in a durable path with acknowledgement and replay rather than a socket queue, which is the distinction drawn in message delivery guarantees.

Why does this only affect some pods? #

Because slow clients are not distributed evenly. A load balancer assigns connections when they arrive, and a pod that happened to receive a batch of mobile users, or that was the target during a network event, keeps them for the life of those connections. This is also why per-pod memory alerts beat fleet averages — see WebSocket observability and monitoring.

Does compression make this better or worse? #

Worse, usually. perMessageDeflate adds roughly 300 KB of zlib state per socket regardless of queue depth, so it raises your baseline everywhere in order to reduce bytes for clients that are mostly not the problem. Enable it only when your payloads are large and repetitive, as covered in permessage-deflate compression trade-offs.

Back to WebSocket Backpressure & Flow Control