Ephemeral port exhaustion on WebSocket proxies #

Your backend servers can hold 200,000 WebSockets each, but the fleet stalls at almost exactly 28,000 connections per backend. New handshakes fail with 502 Bad Gateway, nginx logs connect() to 10.0.1.11:8080 failed (99: Cannot assign requested address), and HAProxy reports Connect() failed: no free ports. The backends are idle and the proxies have spare CPU. The limit is arithmetic: a reverse proxy opens an outbound TCP connection to the backend for every client WebSocket, and each outbound connection from one source address to one destination address and port needs its own local port. Linux hands out about 28,000 of those by default. Long-lived WebSockets never give them back.

Root cause #

A TCP connection is identified by the four-tuple (source IP, source port, destination IP, destination port). When nginx proxies a WebSocket to 10.0.1.11:8080, three of those four are fixed: the proxy’s own IP, the backend’s IP and the backend’s port. Only the source port varies, and the kernel picks it from the ephemeral range, net.ipv4.ip_local_port_range, which defaults to 32768 60999 — 28,232 ports. Once every port in the range is in use toward that one destination, the next connect() fails with EADDRNOTAVAIL.

HTTP proxies rarely hit this, because upstream connections are short-lived or pooled with keepalive: thousands of requests share a few hundred connections. A proxied WebSocket holds its upstream connection for the lifetime of the client connection, one-to-one, so the proxy needs as many simultaneous upstream connections as it has clients. The limit is per destination tuple, which is why adding backends helps and adding proxy CPU does not.

Why each client consumes a proxy port Each client WebSocket makes the proxy open an upstream connection from a new ephemeral source port to the same backend address; after 28,232 ports the next connect fails. Why each client consumes a proxy port Client 1 Client 2 Proxy 10.0.0.5 Backend 10.0.1.11:8080 wss connect src port 32768 wss connect src port 32769 … 28,232 ports later connect() EADDRNOTAVAIL Source IP, destination IP and destination port are fixed — only the source port varies
One destination tuple, one ephemeral range, one hard ceiling.

Resolution #

There are four fixes, and they multiply: each widens a different part of the four-tuple. Start with the port range, then add destination tuples, then source addresses.

# 1. Widen the ephemeral range on the proxy host (/etc/sysctl.d/92-proxy-ports.conf).
#    Leave ports below 1024 alone, and reserve any port a local service listens on.
net.ipv4.ip_local_port_range = 1024 65535          # ~64,500 ports per destination tuple
net.ipv4.ip_local_reserved_ports = 8080,9100       # never handed out as ephemeral
# 2. More destination tuples: listen on several ports per backend, list each one.
upstream realtime {
    hash $arg_cid consistent;
    server 10.0.1.11:8080;
    server 10.0.1.11:8081;          # same process, second listening port
    server 10.0.1.12:8080;
    server 10.0.1.12:8081;
}

# 3. More source addresses: bind upstream connections across several proxy IPs.
split_clients "$remote_addr$remote_port" $proxy_src {
    50%     10.0.0.5;
    *       10.0.0.6;               # a secondary IP on the proxy's interface
}

server {
    location /ws/ {
        proxy_pass http://realtime;
        proxy_bind $proxy_src;      # each source IP gets its own full port range
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

The capacity per proxy toward the backend pool becomes, roughly, ports in range × source IPs × backend IP:port pairs. With the widened range, two source IPs and four destination tuples, that is about 516,000 upstream connections — a comfortable margin for most proxies, and more than their memory will hold.

The fourth fix is architectural: remove the proxy hop from the connection. A layer-4 load balancer that forwards packets (AWS NLB, a Kubernetes Service in IPVS mode with direct routing, or HAProxy in mode tcp with transparent proxying) does not open its own upstream connections at all, so no ephemeral ports are consumed. You give up layer-7 features such as path routing and header-based affinity at that hop, which is often acceptable for a dedicated WebSocket endpoint. The trade-offs are discussed in HAProxy WebSocket load balancing configuration.

In HAProxy, the equivalent of proxy_bind is the source keyword on each server line, which accepts an address and, with a port range, lets you spread connections across several source IPs explicitly.

Maximum upstream WebSockets per backend host With the default ephemeral range one source and one destination allow about twenty-eight thousand connections; widening the range and adding source and destination addresses multiplies the ceiling to over two hundred and fifty thousand. Maximum upstream WebSockets per backend host thousands, by port range and address pairs 1 src × 1 dst 1 src × 2 dst 2 src × 2 dst 0k 100k 200k 300k Default range 1024–65535
Each extra source IP or destination port adds a whole ephemeral range.

Edge cases #

TIME_WAIT makes it worse on churn. When the proxy closes an upstream connection first, that port sits in TIME_WAIT for 60 seconds and cannot be reused toward the same destination. During a mass reconnect — a deploy, a network blip — closed ports and new connections overlap and the effective range shrinks. net.ipv4.tcp_tw_reuse = 1 allows reuse of TIME_WAIT ports for outbound connections and is safe on the proxy side.

Kubernetes Services hide the backend addresses. If nginx proxies to a single ClusterIP, every upstream connection shares one destination tuple, and kube-proxy’s NAT does not help — the tuple is checked on the proxy host before translation. Proxy to pod endpoints directly (as ingress-nginx does) or use a headless Service so each pod is a separate destination.

Conntrack also counts these flows. On a proxy host with netfilter, each client connection and each upstream connection is a tracked flow — two per WebSocket. Size nf_conntrack_max accordingly, as covered in tuning Linux TCP for a million WebSockets.

Verification #

Count upstream connections per destination from the proxy host, and compare against the range:

# Established upstream connections from this proxy, grouped by backend ip:port.
ss -tn state established '( dport = :8080 or dport = :8081 )' \
  | awk 'NR>1 {print $4}' | sort | uniq -c | sort -rn
# The configured range and its size.
sysctl net.ipv4.ip_local_port_range
# Errors that prove exhaustion rather than a backend failure.
grep -c "Cannot assign requested address" /var/log/nginx/error.log

Any destination whose count sits just under the range size is saturated. After the fix, run a load test that holds more connections through one proxy than the old ceiling, and confirm there are no EADDRNOTAVAIL errors — the scripts in load testing WebSockets with k6 can hold connections open at a fixed count.

Fixes compared Four fixes for ephemeral port exhaustion compared by ceiling gain, effort and trade-offs: widening the port range, adding backend ports, adding proxy source IPs and switching to a layer-4 balancer. Fixes compared Ceiling gain Effort Trade-off Widen port range about 2.3× one sysctl reserve listening ports More backend ports × ports per host app + config more listeners More proxy source IPs × IPs network config IP management Layer-4 balancer removes limit architecture no L7 routing here The first three multiply together; the fourth changes the architecture
Start with the sysctl, then add tuples; move to layer 4 when the proxy hop is no longer earning its keep.

Operational checklist #

FAQ #

Why does my proxy fail at about 28,000 WebSocket connections? #

That is the size of Linux’s default ephemeral port range, 32768–60999. Each proxied WebSocket holds one upstream connection, and each upstream connection to the same backend address and port needs a distinct local port.

Does the backend server run out of ports too? #

No. The backend accepts connections on one listening port, and connections are distinguished by the remote (proxy) address and port. The constraint is on the side that initiates connections — the proxy.

Will HTTP/2 or keepalive to the upstream fix it? #

Keepalive helps ordinary HTTP requests share connections, but an upgraded WebSocket occupies its upstream connection exclusively. Multiplexing WebSockets over HTTP/2 to the backend (RFC 8441) is possible with some proxies and servers, but support is limited; adding tuples is the dependable fix.

Is this a problem with managed cloud load balancers? #

Managed balancers handle source-port allocation across their own fleets and large address pools, so you rarely see it there. It appears on self-managed proxy tiers — nginx, HAProxy, Envoy — and on NAT gateways in front of clients that open many outbound WebSockets.

Back to Connection Limits & OS Tuning.