HAProxy WebSocket Load Balancing Configuration #

Connections work for exactly 50 seconds and then die with code 1006, every time, on every client. Or the upgrade returns 200 OK with an HTML body instead of 101 Switching Protocols. Or everything works until you reload HAProxy for an unrelated change and 40,000 sockets drop at once. All three are configuration, and all three have precise fixes.

HAProxy handles WebSockets well — better than most reverse proxies — but only once you tell it that these connections are tunnels, not requests, and that its default timeouts were written for something else entirely.

Root cause #

HAProxy in HTTP mode is built around the request/response cycle. It reads a request, forwards it, reads a response, and applies timeout client and timeout server to each phase. A WebSocket breaks that model at the 101 response: after the upgrade there are no more requests, and a connection can legitimately sit silent for minutes.

Three defaults then work against you:

timeout tunnel is unset, so HAProxy falls back to the smaller of timeout client and timeout server for the tunnelled connection. With the common timeout client 50s, every WebSocket dies at 50 seconds of silence — which is exactly the symptom above, and why it looks like a heartbeat problem when it is not.

timeout http-keep-alive and timeout http-request still apply to the upgrade request itself, so a slow client can be cut off mid-handshake.

Reloads are not seamless by default. A plain systemctl reload starts a new process and stops the old one; without the right expose-fd and hard-stop settings, established tunnels are severed rather than allowed to drain.

Resolution #

The configuration below routes WebSocket traffic explicitly, keeps tunnels alive for an hour, and pins each client to a backend so per-connection state stays valid.

global
    log stdout format raw local0
    # Let a reload inherit listening sockets so no connection is refused mid-swap
    stats socket /var/run/haproxy.sock mode 660 level admin expose-fd listeners
    # Old processes keep serving established tunnels for this long after a reload
    hard-stop-after 30m

defaults
    mode http
    log global
    option httplog
    option http-server-close
    timeout http-request 5s        # the upgrade request itself must arrive quickly
    timeout connect      5s
    timeout client       50s       # plain HTTP only — superseded by tunnel after 101
    timeout server       50s
    timeout tunnel       1h        # THE important one: governs the live WebSocket
    timeout client-fin   30s       # do not hold half-closed connections forever
    timeout tunnel-fin   30s

frontend web
    bind :443 ssl crt /etc/haproxy/certs/site.pem alpn h2,http/1.1

    # Detect a genuine upgrade: the Connection header is a comma-separated list,
    # so match on the token rather than the whole value.
    acl is_upgrade  hdr(Upgrade) -i websocket
    acl is_conn_upg hdr_reg(Connection) -i (^|,)[\s]*Upgrade([\s]*(,|$))
    acl is_ws_path  path_beg /ws

    use_backend ws_nodes if is_upgrade is_conn_upg
    use_backend ws_nodes if is_ws_path
    default_backend http_nodes

backend ws_nodes
    balance leastconn                       # NOT roundrobin: connections are long-lived
    # Sticky routing so per-connection state on a node stays reachable across
    # the upgrade and any HTTP calls the client makes alongside it.
    cookie SRVID insert indirect nocache httponly secure samesite=Lax
    option httpchk GET /healthz
    http-check expect status 200

    # Pass the real client through; the app needs these for origin checks and logs.
    http-request set-header X-Forwarded-Proto https
    http-request set-header X-Forwarded-For %[src]

    # Drain gracefully: a server in MAINT keeps existing tunnels, refuses new ones.
    default-server check inter 5s fall 3 rise 2 on-marked-down shutdown-sessions

    server ws1 10.0.1.11:8080 cookie ws1 maxconn 30000
    server ws2 10.0.1.12:8080 cookie ws2 maxconn 30000
    server ws3 10.0.1.13:8080 cookie ws3 maxconn 30000

Two lines deserve emphasis. balance leastconn is correct for WebSockets in a way that round-robin is not: round-robin distributes connection events evenly, which after any restart or network event means the node that was down receives nothing while the others stay loaded. Least-connections distributes the thing that actually costs you — concurrent sockets.

on-marked-down shutdown-sessions is a deliberate choice, not a default. It means that when a health check fails, HAProxy severs existing tunnels to that server rather than leaving clients attached to a node that has stopped working. Combined with jittered client reconnects, that is what you want; without jitter, it is a stampede, so make sure the client side follows exponential backoff with jitter first.

The leastconn choice is easiest to see after a node comes back. Round-robin hands the recovered node an equal share of new connections, which — because nobody is disconnecting — leaves it near-empty for hours, while least-connections drains the imbalance immediately.

Verification #

Check the upgrade, the idle behaviour, and the reload separately.

# Does the upgrade actually reach the backend and return 101?
curl -i -N -H "Connection: Upgrade" -H "Upgrade: websocket" \
     -H "Sec-WebSocket-Version: 13" -H "Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==" \
     https://example.com/ws

# Live connection counts per backend server, and which are draining
echo "show stat" | socat /var/run/haproxy.sock stdio | cut -d, -f1,2,5,18

# Reload without dropping tunnels, then confirm old processes are still serving
systemctl reload haproxy && pgrep -a haproxy | head

The idle test is the one people skip: open a connection, send nothing for three minutes, and confirm it is still alive. If it dies near your timeout client value, timeout tunnel is not being applied — usually because it was set in a backend that the traffic is not actually reaching, which the show stat output will reveal.

Operational checklist #

• Set timeout tunnel explicitly to at least an hour; without it HAProxy falls back to the smaller of client and server.
• Keep timeout tunnel longer than your application heartbeat interval, so the heartbeat — not the proxy — decides when a socket dies.
• Route on the Upgrade header token rather than the whole Connection value, which is a comma-separated list.
• Use balance leastconn, not round-robin, so long-lived connections spread by concurrency rather than arrival.
• Insert a sticky cookie if any per-connection state lives in the node process.
• Enable expose-fd listeners and set hard-stop-after so reloads drain instead of severing.
• Forward X-Forwarded-Proto and X-Forwarded-For, which origin checks and logs both depend on.
• Rollback: keep the previous config in version control and validate with haproxy -c -f before every reload.

FAQ #

Do I need mode tcp instead of mode http? #

No, and you usually do not want it. HTTP mode gives you path routing, header inspection, sticky cookies and useful logs, and it handles the upgrade correctly once timeout tunnel is set. TCP mode is the right answer only when you must pass traffic HAProxy cannot parse — for instance terminating TLS elsewhere and forwarding opaque bytes — and it costs you every routing feature above.

Why do connections still drop at 50 seconds after I set the timeout? #

Almost always because timeout tunnel was set in defaults but the traffic is served by a backend that overrides timeout server, or because it was set in the wrong section entirely. Check show stat to confirm which backend the connections land in, and remember that a proxy in front of HAProxy — a cloud load balancer, a CDN — has its own idle timeout that HAProxy cannot override.

How do sticky cookies interact with the WebSocket upgrade? #

The cookie is set on an earlier HTTP response and sent by the browser with the upgrade request, so affinity is established before the socket exists. That is why cookie stickiness works for WebSockets at all. If the client connects to wss:// as its very first request with no prior page load, there is no cookie yet — fall back to source-based balancing or issue the cookie from the page that opens the socket, as discussed in load balancer sticky sessions.

Is HAProxy’s health check enough to detect a broken node? #

For process liveness, yes. For WebSocket-specific failures — a node that accepts TCP but whose upgrade handler is wedged — an HTTP check on /healthz may pass while sockets fail. Make the health endpoint assert something real: that the WebSocket server is listening, that the fan-out bus is reachable, and that the node is below its connection budget.

How does this compare to nginx for WebSockets? #

Both work. HAProxy has better connection-level observability (show stat is unmatched), more precise timeout control, and genuinely seamless reloads. nginx is more common where the same server also terminates TLS for a static site and is covered in configuring nginx for WebSocket upgrades. The failure modes are near-identical; only the directive names differ.

Related #

• Load Balancer Sticky Sessions — when affinity is required and when it is a smell.
• Configuring AWS ALB for WebSocket Sticky Sessions — the managed equivalent of this configuration.
• Configuring nginx for WebSocket Upgrades — the same problem in the other common proxy.
• Draining WebSocket Connections During Deploys — what on-marked-down and hard-stop-after are protecting.
• Connection Lifecycle & Heartbeats — keeping the tunnel busy enough that no timeout ever fires.

Back to Load Balancer Sticky Sessions