Handling WebSocket bufferedAmount Backpressure #

You are broadcasting to 20,000 sockets and one node’s RSS climbs steadily while the others sit flat. Heap snapshots point at retained buffers held by socket send queues, and the pattern is always the same handful of clients: mobile users on a weak connection, or a laptop that went to sleep with the tab open. socket.send() returned without complaint every single time, because send() does not fail when the peer is slow — it queues. This page is about the number that tells you it happened, bufferedAmount, and what to do the moment it starts climbing.

Root cause #

send() is fire-and-forget by design. Underneath it, TCP flow control is doing exactly its job: the client’s receive window shrinks as the application fails to read, the kernel stops sending, and the sender’s socket buffer fills. At that point Node cannot push any more bytes into the kernel, so the ws sender keeps them in userspace — an ordinary JavaScript array of buffers with no bound at all. bufferedAmount is the size of that userspace queue plus anything pending in the underlying socket.

The critical asymmetry: every other queue in the chain is bounded. The client’s receive buffer is bounded, the kernel send buffer is bounded by SO_SNDBUF, the network is bounded by physics. Only the userspace queue is unbounded, so it is the one that absorbs the difference between what you produce and what the client consumes — permanently, at a rate of exactly (produce − drain) bytes per second, until the process dies or the socket does.

Once you see it this way, the fix follows: read the queue depth before every send and let its value change your behaviour. Nothing else can, because nothing else in the stack knows what your messages mean.

High and low watermarks with hysteresis Queue depth rises past the high-water mark, which pauses non-critical sends; sending only resumes after the queue drains below the low-water mark, and a kill mark closes the socket entirely. Queue depth over time, with the three marks kill mark 8 MB — close 1013 high mark 2 MB — pause + coalesce low mark 256 KB — resume pause resume + flush pause again The gap between the blue and teal lines is what stops the socket flapping on every drained packet
Send-buffer growth when the producer outruns the socket Buffered bytes over 20 seconds when 600 messages per second are offered to a socket draining 300 per second: unbounded growth versus a policy that caps the queue at the high-water mark. Send-buffer growth when the producer outruns the socket 600 msg/s offered, 300 msg/s drained, 400 B each No backpressure Watermark policy 0 MB 1 MB 2 MB 3 MB 0s 5s 10s 15s 20s
300 undrained messages per second at 400 bytes is 120 KB/s of permanent growth per slow socket — the watermark policy turns the line into a plateau.

Resolution #

Route every outbound write through one function that reads the queue before deciding. The hysteresis pair — pause high, resume low — is what keeps it from thrashing, and the pending map is what makes resuming cheap: instead of replaying a backlog of stale state, you send the current value of each key once.

import { WebSocket } from 'ws';

const HIGH_WATER_MARK_BYTES = 2 * 1024 * 1024;
const LOW_WATER_MARK_BYTES  = 256 * 1024;
const KILL_WATER_MARK_BYTES = 8 * 1024 * 1024;
const STALL_SWEEPS_BEFORE_TERMINATE = 2;   // heartbeat sweeps with zero drain

interface Guarded {
  socket: WebSocket;
  paused: boolean;
  pending: Map<string, unknown>;   // last value wins, per key
  lastQueue: number;
  stalledSweeps: number;
  dropped: number;
}

export function send(g: Guarded, key: string, value: unknown, critical = false): boolean {
  if (g.socket.readyState !== WebSocket.OPEN) return false;
  const queued = g.socket.bufferedAmount;

  if (queued > KILL_WATER_MARK_BYTES) {
    // Beyond catching up: a reconnect and a fresh snapshot is cheaper for both sides.
    g.socket.close(1013, 'client too slow');
    return false;
  }

  if (queued > HIGH_WATER_MARK_BYTES) {
    g.paused = true;
    if (!critical) { g.pending.set(key, value); g.dropped += 1; return false; }
  } else if (g.paused && queued < LOW_WATER_MARK_BYTES) {
    g.paused = false;
    // Flush the collapsed state in one pass. This is why coalescing beats queueing:
    // 40,000 buffered ticks become one message per key.
    for (const [k, v] of g.pending) g.socket.send(JSON.stringify({ k, v }));
    g.pending.clear();
  }

  g.socket.send(JSON.stringify({ k: key, v: value }));
  return true;
}

/** Run on the same interval as your heartbeat sweep. */
export function auditQueue(g: Guarded): void {
  const queued = g.socket.bufferedAmount;
  // A queue that has not moved at all between sweeps is not slow, it is stuck:
  // close() would wait for a drain that will never happen, so terminate instead.
  if (queued > 0 && queued >= g.lastQueue) g.stalledSweeps += 1;
  else g.stalledSweeps = 0;
  g.lastQueue = queued;

  if (g.stalledSweeps >= STALL_SWEEPS_BEFORE_TERMINATE) g.socket.terminate();
}

The auditQueue half matters as much as the send guard. A client whose TCP connection is black-holed — a laptop lid closed on a train, a NAT that dropped the mapping — never drains and never errors, so the queue simply freezes at whatever it was. close() on that socket politely waits for a closing handshake that cannot arrive; only terminate() reclaims the memory. Pair it with the ping/pong sweep from implementing WebSocket ping/pong in Node.js so you are already iterating the client set on a timer.

Verification #

Reproduce a slow consumer deliberately rather than waiting for one. Connect a client that completes the upgrade and then never reads from the socket — the receive window closes within a second or two and your queue starts climbing, exactly as a real stalled client would.

# Per-socket send queue depth from the OS side (Send-Q, bytes)
ss -tn state established '( sport = :8080 )' | sort -k3 -nr | head

# Watch the guarded gauge react: it should plateau, not climb
curl -s localhost:9090/metrics | grep -E 'ws_send_queue_bytes|ws_messages_dropped_total'

Two signals confirm the policy is live. Queue depth plateaus near the high-water mark instead of growing without bound, and the drop counter becomes non-zero and steady. Drops that accelerate while depth stays flat mean the coalescing map is doing the work and the client is simply too slow for the feed; depth that climbs with zero drops means the guard is not on the hot send path at all.

Decision tree for one outbound message Check the socket is open, then the kill mark, then the high-water mark; critical messages always send, stateful messages coalesce by key and ephemeral messages are dropped and counted. send() read queue > kill mark? close 1013 > high mark? by priority critical — always send stateful — coalesce by key ephemeral — drop, count below high mark — send no no

Operational checklist #

FAQ #

What is a good bufferedAmount threshold? #

There is no universal number, because the right threshold is a duration, not a size. Take your per-client output rate in bytes per second and multiply by the seconds of backlog you can tolerate: at 40 KB/s and a three-second tolerance, that is 120 KB. For most dashboards the answer lands between 256 KB and 2 MB. Anything that gives you more than about five seconds of backlog is too generous to be useful, because the user has been staring at stale data the whole time.

Does bufferedAmount include the WebSocket frame headers? #

In the ws package, yes — it reflects encoded bytes including framing, which is what you want since that is what actually occupies memory. In browsers, the HTML standard defines it as the bytes of the messages you queued, and implementations vary slightly on whether framing overhead is counted. Either way, do not use it for precise accounting; use it as a threshold signal.

Why does bufferedAmount stay at zero even though the client is clearly behind? #

Because the bottleneck is not the network. If your serialisation, encryption or application logic is the constraint, bytes never reach the socket in the first place and the queue stays empty while the event loop falls behind. Watch monitorEventLoopDelay from perf_hooks alongside the queue: high lag with an empty queue points at CPU, and the fix is cheaper encoding or sharding, not watermarks.

Does this work behind AWS ALB or nginx? #

Yes, with one nuance: the proxy terminates the client connection and opens its own to your server, so the peer whose receive window you are pushing against is the proxy. Proxies buffer a bounded amount and then stop reading, which produces exactly the same growth in your userspace queue. Thresholds and policy are unchanged; what changes is that a slow client and a saturated proxy look identical from your side, so tag the metric with the upstream if you run more than one.

What about perMessageDeflate? #

Compression makes the number harder to read. Bytes waiting in the deflate pipeline count towards the queue before they are compressed, so a client can look substantially worse than it is, and zlib itself holds roughly 300 KB of state per socket. Pick thresholds with compression in whatever state you actually ship, and if you are enabling it purely to reduce backpressure, measure first — see permessage-deflate compression trade-offs.

Back to WebSocket Backpressure & Flow Control