Disconnecting slow WebSocket consumers #

One client on a congested hotel Wi-Fi subscribes to a busy channel. It reads at a fraction of the rate you publish, its outbound queue on the server grows by megabytes a minute, and eventually the Node process pauses for garbage collection long enough that every other client on the node sees a latency spike. Watermarks and coalescing slow the growth, but some consumers will never catch up — and keeping them connected is worse for them and for everyone else than disconnecting them. This page defines a policy for when to give up on a slow consumer, how to close so the client recovers correctly, and how to stop the same client from immediately repeating the cycle.

Root cause #

A server that publishes faster than a client reads has only three choices for each outbound message: queue it, drop it, or stop producing for that client. Queuing without bound is the default in ws and every other WebSocket library — send() never refuses — so memory grows until the process falls over. Dropping silently corrupts state for event streams. Pausing production is ideal for request/response flows but impossible for a broadcast channel whose other subscribers need the data now.

For broadcast workloads the honest answer is a fourth choice: evict the consumer and let it resynchronize. A client that is 30 seconds behind on a live feed is not helped by receiving 30 seconds of stale updates in order; it is helped by dropping them and loading a fresh snapshot. Eviction converts an unbounded, fleet-harming queue into one reconnect and one snapshot, which is cheaper for the server and gives the user current data sooner.

Consumer health states A consumer moves from healthy to lagging when its buffer passes a soft mark, to degraded when the lag persists, and is evicted at a hard buffer mark or maximum message age; a lagging consumer that drains returns to healthy. Consumer health states Healthy buffer drains Lagging above soft mark Degraded coalesce, skip non-critical Evicted close 1013, resync soft mark lag persists hard mark or age drained Eviction is a controlled path to resync, not a punishment — the client comes back to fresh state
Most slow clients recover at the lagging stage; eviction handles the ones that cannot.

Resolution #

Track two signals per socket: bytes buffered (bufferedAmount) and the age of the oldest unsent message. Bytes protect memory; age protects correctness, because a small buffer of very old messages is just as useless to the user as a large one. The policy below evicts on either signal, after a grace period so a momentary stall does not trigger it.

import { WebSocket } from 'ws';

const SOFT_BUFFER_BYTES = 512 * 1024;       // start degrading: skip non-critical streams
const HARD_BUFFER_BYTES = 4 * 1024 * 1024;  // evict immediately above this
const MAX_LAG_MS = 15_000;                  // oldest undelivered message may be this old
const GRACE_MS = 5_000;                     // continuous lag before evicting on age
const CLOSE_TRY_AGAIN_LATER = 1013;
const CHECK_INTERVAL_MS = 1_000;

interface Consumer {
  ws: WebSocket;
  outbox: { enqueuedAt: number; payload: string; critical: boolean }[];
  laggingSince: number | null;
}

export function send(c: Consumer, payload: string, critical = false) {
  if (c.ws.bufferedAmount > SOFT_BUFFER_BYTES && !critical) {
    metrics.skippedNonCritical.inc();       // degrade: shed optional traffic first
    return;
  }
  c.outbox.push({ enqueuedAt: Date.now(), payload, critical });
  drain(c);
}

function drain(c: Consumer) {
  while (c.outbox.length && c.ws.bufferedAmount < SOFT_BUFFER_BYTES) {
    c.ws.send(c.outbox.shift()!.payload);
  }
}

export function checkConsumer(c: Consumer, now = Date.now()) {
  drain(c);
  const buffered = c.ws.bufferedAmount;
  const oldest = c.outbox[0]?.enqueuedAt;
  const lagMs = oldest ? now - oldest : 0;

  if (buffered > HARD_BUFFER_BYTES) return evict(c, 'buffer_hard_limit');
  if (lagMs > MAX_LAG_MS) {
    c.laggingSince ??= now;
    if (now - c.laggingSince > GRACE_MS) return evict(c, 'lag_age');
  } else {
    c.laggingSince = null;                  // recovered: reset the grace clock
  }
}

function evict(c: Consumer, reason: string) {
  metrics.slowConsumerEvictions.inc({ reason });
  c.outbox.length = 0;                      // free the queue now, not after close completes
  // 1013 = "try again later": the client should reconnect with backoff and resync.
  c.ws.close(CLOSE_TRY_AGAIN_LATER, 'slow consumer');
  // If the close frame itself cannot be flushed, do not wait for it.
  setTimeout(() => c.ws.terminate(), 2_000).unref();
}

// setInterval(() => consumers.forEach((c) => checkConsumer(c)), CHECK_INTERVAL_MS);

The close code matters. 1013 Try Again Later tells a well-behaved client that the server is fine but the connection should be retried after a delay, and — paired with a reason string — that the client should discard its local stream position and resync from a snapshot rather than attempting to resume. Resuming would ask the server to replay exactly the backlog that caused the eviction. The client-side resync is covered in reconciling snapshots and deltas over WebSockets.

To stop the same client immediately repeating the cycle, remember recent evictions by client identifier and offer a degraded subscription on reconnect: coalesced updates only, or a lower update rate. A client evicted three times in ten minutes is on a network that cannot carry the full stream, and serving it a lighter one is a better experience than a reconnect loop — see coalescing high-frequency WebSocket updates.

Heap budget per node before any backlog Estimated heap for 5,000 to 100,000 concurrent sockets at 34 kilobytes per socket plus 8 kilobytes of application state, against a 1024 megabyte pod limit. Heap budget per node before any backlog 34 KB socket + 8 KB app state each — dashed line is a 1024 MB pod limit Socket + buffers App state 5.0k 205 MB 332 MB 10k 410 MB 830 MB 25k 1.0k MB 1.7k MB 391 MB 50k 2.1k MB 1024 MB pod limit
The baseline cost per socket is small; one slow consumer holding 4 MB of backlog costs as much as a hundred healthy connections.

Verification #

Simulate a slow reader rather than waiting for one. A test client can pause its socket so the kernel stops acknowledging data:

// Test client: connect, subscribe to a busy channel, then stop reading.
import WebSocket from 'ws';
const ws = new WebSocket('ws://localhost:8080/?channel=firehose');
ws.on('open', () => {
  // Pausing the underlying socket stops reads; the server's buffer will grow.
  (ws as any)._socket.pause();
});
ws.on('close', (code, reason) => console.log('closed', code, String(reason)));
// Expect: "closed 1013 slow consumer" after MAX_LAG_MS + GRACE_MS or at the hard limit.

On the server, graph slow_consumer_evictions_total by reason alongside process heap. Eviction counts should be a tiny fraction of connections; if they are large, your publish rate or message size is wrong for your audience, and eviction is hiding a design problem. The heap graph should stop showing sawtooth growth that correlates with individual clients.

Eviction and resync The server evicts a lagging client with close code 1013; the client discards its position, reconnects after backoff, loads a snapshot and receives a coalesced stream. Eviction and resync Client Server Snapshot API lag > 15 s for 5 s close 1013 slow consumer discard stream position reconnect after backoff fetch snapshot coalesced stream resumes Resuming from the old position would replay the very backlog that caused the eviction
Evict, snapshot, continue — the client ends up current faster than it would have by waiting.

Operational checklist #

FAQ #

Isn’t disconnecting users a bad experience? #

Less bad than the alternative. A client that is 30 seconds behind shows stale data and, if left alone, eventually degrades the whole node. An eviction followed by a snapshot shows current data within a second or two, and a subtle “reconnecting” indicator is enough for the user to understand.

Which close code should I use for a slow consumer? #

1013 Try Again Later fits best: it signals a temporary server-side condition and tells clients to retry after a delay. 1008 Policy Violation is sometimes used, but many clients treat it as permanent and stop reconnecting.

How do I tell a slow client from a dead one? #

A dead client stops acknowledging entirely; its buffer grows and never drains, and it also stops answering pings. A slow one drains, just not fast enough. The half-open detection sweep catches the first; this policy catches the second.

Should limits be per message size or per byte? #

Per byte for memory protection, because one large message can matter more than a hundred small ones. Use message age, not message count, for the correctness limit.

Back to WebSocket Backpressure & Flow Control.