Detecting half-open WebSocket connections #

Your connection gauge says 42,000 clients are online, but your analytics say 31,000. Messages to some users “send” successfully and are never seen. Memory creeps up by a few megabytes an hour and resets only on deploy. These are the symptoms of half-open connections: sockets the server still believes are established while the peer has long since disappeared — a phone that lost signal in a lift, a laptop that went to sleep, a NAT box that rebooted. Nothing on the server fails, because nothing on the server has tried to prove the peer is still there. This page explains why TCP lets these sockets live indefinitely and builds a detector that finds them within a bounded window.

Root cause #

TCP learns that a peer is gone in exactly two ways: it receives a FIN or RST, or it tries to deliver data and retransmits until it gives up. A peer that vanishes without sending anything — power loss, radio loss, a middlebox silently dropping the flow — produces neither signal. If your server is also not sending, the connection is simply quiet, and quiet is a perfectly legal TCP state. The kernel keeps the socket in ESTABLISHED, your ws instance keeps its readyState at OPEN, and your registry keeps counting it.

Even when the server does send, detection is slow. The first write succeeds immediately because it only copies bytes into the kernel send buffer. TCP then retransmits with exponential backoff, and on Linux the default tcp_retries2 = 15 means the kernel keeps trying for roughly 15 minutes before declaring the connection dead and surfacing an error. During that time bufferedAmount grows, the socket buffer fills, and — if you ignore backpressure — your process heap grows with it, the failure covered in fixing slow-consumer memory growth.

How a half-open socket forms A client loses its network without sending a FIN or RST; the server kernel keeps the socket established and retransmits for about fifteen minutes while the Node process still believes the socket is open. How a half-open socket forms Client NAT / radio Server kernel Node process messages flow normally device loses network no FIN, no RST sent send() copies to buffer retransmit, backoff ~15 min before error readyState still OPEN Without an application-level deadline, the only thing that ever closes this socket is TCP's retransmission limit
The server finds out only when it tries to prove the peer exists — and TCP takes its time.

A half-open socket is distinct from the CLOSE_WAIT leak, where the peer did send a FIN and your code never closed its side; that one is covered in fixing WebSocket CLOSE_WAIT accumulation. Here nobody sent anything, which is why the fix has to be an active probe with a deadline.

Resolution #

The detector below combines two signals. A liveness deadline records when the connection last produced any inbound bytes — a pong, a message, anything — and terminates it when the silence exceeds a bound. A write probe watches bufferedAmount: if bytes queued for a client stop draining across several checks, the peer is not reading, whether it is dead or merely very slow, and either way the socket is costing memory for no benefit.

import { WebSocketServer, WebSocket } from 'ws';

const PING_INTERVAL_MS = 20_000;       // how often we solicit proof of life
const LIVENESS_DEADLINE_MS = 45_000;   // silence tolerated before termination
const STALL_CHECKS_BEFORE_KILL = 3;    // consecutive checks with a non-draining buffer
const STALL_MIN_BYTES = 64 * 1024;     // ignore tiny residues that drain on their own

interface Probed extends WebSocket {
  lastInboundAt: number;
  lastBuffered: number;
  stalledChecks: number;
}

const wss = new WebSocketServer({ port: 8080 });

wss.on('connection', (raw) => {
  const ws = raw as Probed;
  ws.lastInboundAt = Date.now();
  ws.lastBuffered = 0;
  ws.stalledChecks = 0;
  const touch = () => { ws.lastInboundAt = Date.now(); };
  ws.on('pong', touch);      // reply to our probe
  ws.on('message', touch);   // any application traffic counts as proof too
  ws.on('ping', touch);      // a non-browser client may ping us
});

function reap(ws: Probed, reason: string) {
  metrics.halfOpenTerminated.inc({ reason });
  ws.terminate();            // destroy the TCP socket; a close handshake would never complete
}

setInterval(() => {
  const now = Date.now();
  for (const client of wss.clients) {
    const ws = client as Probed;

    // Signal 1: nothing received from the peer for too long.
    if (now - ws.lastInboundAt > LIVENESS_DEADLINE_MS) {
      reap(ws, 'liveness_deadline');
      continue;
    }

    // Signal 2: bytes are queued but not draining — the peer is not ACKing.
    const buffered = ws.bufferedAmount;
    const draining = buffered < ws.lastBuffered || buffered < STALL_MIN_BYTES;
    ws.stalledChecks = draining ? 0 : ws.stalledChecks + 1;
    ws.lastBuffered = buffered;
    if (ws.stalledChecks >= STALL_CHECKS_BEFORE_KILL) {
      reap(ws, 'send_stalled');
      continue;
    }

    ws.ping();               // solicit a pong for the next sweep
  }
}, PING_INTERVAL_MS);

Two choices are deliberate. The deadline is measured from the last inbound byte of any kind, not only pongs, so a chatty client never gets killed because a single pong was delayed behind a large message. And termination uses terminate() rather than close(): a close handshake needs the peer to answer, which a half-open peer by definition cannot, so close() would just add another 30 seconds of waiting before ws gives up.

The two signals catch different cases, and neither is enough alone.

Which signal catches which failure A comparison of liveness deadlines, write-stall probes and TCP keepalive against four failure cases. Which signal catches which failure Liveness deadline Write probe TCP keepalive Idle peer vanished caught in ≤45 s nothing queued hours by default Busy peer vanished caught in ≤45 s caught in ~60 s hours by default Alive but not reading pongs may still flow caught never Healthy slow network survives survives if draining survives Default Linux keepalive waits two hours before its first probe, which is why it is not a detector here
Deadline for dead peers, write probe for peers that stopped reading.

If you want the kernel to help as well, you can shorten TCP’s own give-up time for this socket with TCP_USER_TIMEOUT, which bounds how long unacknowledged data may sit before the kernel errors the connection. Node does not expose the option directly, but the application-level deadline above makes it unnecessary for most deployments.

Verification #

Reproduce a half-open peer deterministically by dropping packets rather than closing the client. On a Linux test host, connect a client, then blackhole its traffic with iptables so no FIN or RST is ever delivered:

# 1. Connect a test client from 10.0.0.25 and leave it idle.
# 2. On the server, silently drop everything from that client.
sudo iptables -I INPUT -s 10.0.0.25 -p tcp --dport 8080 -j DROP
# 3. Watch the socket: it stays ESTABLISHED, with retransmits climbing.
watch -n 5 "ss -tnoi state established '( sport = :8080 )' dst 10.0.0.25"
# 4. After LIVENESS_DEADLINE_MS the detector terminates it; remove the rule.
sudo iptables -D INPUT -s 10.0.0.25 -p tcp --dport 8080 -j DROP

The timer:(on,...) field in ss -o output shows the retransmission timer ticking; once the detector fires, the socket leaves the list. In production, compare the server’s connection gauge against an independent count — distinct users with a message received in the last minute. A persistent gap between the two is the population of half-open sockets, and it should shrink to near zero once the detector ships.

The timeline below shows what the detector bounds: the window between the peer dying and the server releasing its resources.

Detection window with a 45 s liveness deadline A peer dies eight seconds after its last pong; pings at twenty and forty seconds go unanswered and the sweep at sixty seconds terminates the socket. Detection window with a 45 s liveness deadline half-open last pong received (0s) peer loses network (8s) ping, no reply (20s) ping, no reply (40s) sweep: deadline passed (60s) terminate() (60s) Worst case is the deadline plus one sweep interval — here about 65 s, versus roughly 15 minutes for TCP alone
The deadline turns an open-ended leak into a bounded one.

Operational checklist #

FAQ #

Why does readyState stay OPEN after the client disconnected? #

readyState reflects what the server has been told. A client that loses its network never sends a close frame or a FIN, so the server has received no signal to change state. It stays OPEN until the server itself probes the connection and a deadline or TCP error closes it.

Isn’t TCP keepalive designed for exactly this? #

It is, but its Linux defaults — first probe after two hours of idle, then nine probes 75 seconds apart — are far too slow for a real-time service, and it only runs when the socket is completely idle. An application-level ping with a deadline gives you a detection window measured in seconds and is independent of kernel tuning.

Will the detector disconnect users on slow mobile networks? #

Only if nothing at all arrives for the whole deadline. A slow network delays pongs by hundreds of milliseconds or a few seconds, which is far inside a 45-second window. If you see false positives, raise the deadline rather than the ping frequency, and check that large outbound messages are not delaying pong processing on the client.

How is this different from CLOSE_WAIT? #

In CLOSE_WAIT the peer did close — the kernel received its FIN — and your application has not closed its side. A half-open connection is the opposite: the peer is gone but sent nothing, so the kernel still reports ESTABLISHED. The fixes differ accordingly: close handling for one, active probing for the other.

Back to Connection Lifecycle & Heartbeats.