Tuning Linux TCP for a million WebSockets #

You raised the descriptor limit, and the node now accepts connections past the old wall. Somewhere between 100,000 and a million idle WebSockets, new problems appear: the kernel logs nf_conntrack: table full, dropping packet, connection bursts after a deploy produce SYN timeouts, dmesg reports TCP: out of memory -- consider tuning tcp_mem, and throughput collapses even though Node’s heap is fine. These limits live in the kernel’s networking stack, and their defaults assume a machine with a few thousand busy connections, not hundreds of thousands of mostly idle ones. This page covers the sysctls that actually matter for WebSocket density, the arithmetic behind each value, and the ones that are commonly changed for no benefit.

Root cause #

Every established TCP connection costs kernel memory outside your process: a socket structure of a couple of kilobytes, plus receive and send buffers that grow when data is in flight. The kernel caps the total with net.ipv4.tcp_mem, measured in pages; when the total passes the pressure threshold, the kernel shrinks buffers, and past the maximum it refuses allocations — the “out of memory” message — even when free RAM is plentiful. With a million sockets, even the minimum 4 KB buffer each adds up to gigabytes.

Two other defaults bite during bursts rather than at steady state. The listen backlog (net.core.somaxconn and your listen() call’s backlog) bounds how many completed handshakes can wait for accept(); during a reconnect storm tens of thousands of clients arrive in a second, the queue overflows, and the kernel drops SYNs, so clients retry after a second, then three, then seven. And if the host runs iptables or nftables with connection tracking — common on Kubernetes nodes — every connection occupies a conntrack entry, and the table’s default size (nf_conntrack_max, often 65,536 or 262,144) is far below a million.

Kernel limits in the order you meet them Kernel limits met while scaling WebSocket connections: descriptor limit, accept backlog, conntrack table size, total TCP memory and per-socket buffer sizes. Kernel limits in the order you meet them nofile (RLIMIT) descriptors per process; fixed first ~1k default somaxconn / backlog handshakes waiting for accept() during storms 4096 default nf_conntrack_max tracked flows on hosts with netfilter state 64k–256k tcp_mem total socket buffer pages; pressure shrinks buffers ~9% RAM tcp_rmem / tcp_wmem per-socket buffer min, default, max per socket Conntrack is the silent one — drops appear only in dmesg
After descriptors, it is queues and memory accounting — not CPU — that stop a dense node.

Resolution #

The values below target a node holding up to a million mostly idle WebSockets with small messages. Treat them as a starting point derived from arithmetic, then verify against your own traffic.

# /etc/sysctl.d/91-websocket-tcp.conf

# --- Accept queue: absorb reconnect storms without SYN drops ----------------
net.core.somaxconn = 65535              # cap on any listen() backlog
net.ipv4.tcp_max_syn_backlog = 65535    # half-open handshakes queued
net.core.netdev_max_backlog = 16384     # packets queued per CPU before the stack

# --- Socket buffer memory ---------------------------------------------------
# Per-socket min / default / max bytes. Small defaults keep idle sockets cheap;
# autotuning still grows busy sockets toward the max.
net.ipv4.tcp_rmem = 4096 16384 4194304
net.ipv4.tcp_wmem = 4096 16384 4194304
# Total TCP buffer memory, in 4 KiB pages: low / pressure / high.
# 1M sockets x ~8 KiB typical in-use buffers ~ 8 GiB ~ 2M pages at "pressure".
net.ipv4.tcp_mem = 1572864 2097152 3145728

# --- Connection tracking (only if netfilter conntrack is loaded) ------------
net.netfilter.nf_conntrack_max = 2097152
# ESTABLISHED entries default to 5 days; keep them past your longest idle socket
# but not forever, so dead flows eventually free their slots.
net.netfilter.nf_conntrack_tcp_timeout_established = 86400

# --- Dead-peer hygiene (backs up application heartbeats) --------------------
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_keepalive_probes = 5

Your server must also ask for a large backlog; somaxconn only caps it. Node’s default backlog is 511. Pass a larger value when listening:

import http from 'node:http';
const LISTEN_BACKLOG = 65_535;   // capped by net.core.somaxconn
const server = http.createServer();
server.listen({ port: 8080, backlog: LISTEN_BACKLOG });

Conntrack deserves a decision rather than a bigger number. If the node does not need stateful firewalling for WebSocket traffic, exempt the port from tracking entirely with a NOTRACK rule in the raw table, which removes the table from the path. On Kubernetes, kube-proxy in iptables mode relies on conntrack for Service NAT, so raising the maximum is the practical option; kube-proxy also sets nf_conntrack_max itself from its --conntrack-max-per-core flag, which will overwrite a sysctl you set by hand.

Kernel socket buffer memory by connection count Kernel buffer memory grows linearly with connection count, from under half a gibibyte at one hundred thousand idle sockets to sixteen gibibytes at a million busy ones. Kernel socket buffer memory by connection count receive + send buffers, excluding socket structures 4 KiB per socket (idle) 16 KiB per socket (busy) 0 GiB 5 GiB 10 GiB 15 GiB 20 GiB 100k sockets 8 GiB 500k sockets 4 GiB 16 GiB 1M sockets
This memory is outside your Node heap — size tcp_mem for it, and count it in the node's memory budget.

What not to change #

Several sysctls appear in every “tune Linux for high concurrency” list and do nothing useful for a WebSocket server. tcp_tw_reuse and the removed tcp_tw_recycle concern outbound connections stuck in TIME_WAIT; a server that accepts connections rarely accumulates TIME_WAIT for its own port, and tcp_tw_recycle actively broke clients behind NAT before it was removed. tcp_fin_timeout controls FIN_WAIT_2, not CLOSE_WAIT, so it will not fix the leak covered in fixing WebSocket CLOSE_WAIT accumulation. And ip_local_port_range matters on proxies making outbound connections, which is covered in ephemeral port exhaustion on WebSocket proxies, not on the server terminating them.

Keepalive settings are a backstop, not a heartbeat. Application-level pings, as in detecting half-open WebSocket connections, detect dead peers in seconds; kernel keepalive only cleans up sockets the application has forgotten.

Verification #

Look for the specific symptoms each setting prevents, before and after the change:

# Accept queue overflows and SYN drops since boot (should stop increasing under load).
nstat -az TcpExtListenOverflows TcpExtListenDrops TcpExtTCPReqQFullDrop
# Current backlog vs limit on the listening socket: Recv-Q is queued, Send-Q is the limit.
ss -ltn 'sport = :8080'
# TCP memory in pages: "mem" vs the tcp_mem thresholds.
cat /proc/net/sockstat
# Conntrack usage vs max, and drops logged by the kernel.
sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max
dmesg -T | grep -E 'nf_conntrack: table full|TCP: out of memory'

Then run a storm test: hold your target idle connection count, drop them all at once (restart the server), and let clients reconnect with jitter. ListenOverflows should not move, and the fleet should be fully reconnected within your backoff window.

Reconnect arrivals after a node restart Reconnect attempts per second for 20000 dropped clients: a fixed retry delay lands every client in one second, full jitter spreads them across 6 seconds. Reconnect arrivals after a node restart 20k clients, 6s window — fixed delay vs full jitter Fixed delay Full jitter 0 5.0k 10k 15k 20k 3.3k 1s 3.3k 2s 3.2k 3s 3.4k 4s 3.4k 5s 3.4k 6s
The accept queue must hold the peak second of this curve; with a backlog of 511 most of it would be dropped and retried with SYN backoff.

Operational checklist #

FAQ #

How many WebSocket connections can one Linux server handle? #

Millions of idle connections are achievable on a large machine; the limit is memory — kernel socket buffers plus your application’s per-connection state — and the CPU cost of whatever traffic those connections carry. The kernel defaults, not the hardware, are what stop most servers far earlier.

Do I need to change ip_local_port_range on the server? #

No. A server accepting connections uses one local port (its listening port) for all of them; connections are distinguished by the client’s address and port. Ephemeral ports matter on machines making outbound connections, such as proxies.

Why do I see “TCP: out of memory” when the machine has free RAM? #

The kernel limits TCP buffer memory separately through tcp_mem, which defaults to a fraction of RAM calculated at boot. Past its maximum, allocations fail regardless of free memory. Raise it in proportion to your connection count.

Should I enable TCP_NODELAY? #

Yes, for interactive WebSocket traffic — and ws already sets it on accepted sockets. Nagle’s algorithm delays small writes to coalesce them, which adds latency to exactly the small, frequent frames real-time apps send.

Back to Connection Limits & OS Tuning.