WebSocket sticky sessions with nginx ip_hash #

Clients on your Socket.IO or SockJS fallback transport get 400 Session ID unknown errors whenever they hit a second backend, and after you add a node, a third of your connected users reconnect at once. Both are stickiness problems in the nginx upstream. A pure WebSocket connection is naturally sticky — once upgraded, the TCP connection stays on the backend it was proxied to — but everything around it is not: the long-polling handshake that precedes an upgrade, the reconnect after a drop, and any HTTP request that expects to find in-memory session state on the same node. This page configures nginx to route a client consistently, and shows where ip_hash stops being good enough.

Root cause #

nginx’s default upstream balancing is round robin: each new request or connection goes to the next server. For a raw WebSocket that is fine, since the connection is one long request. It breaks in two situations. First, transports that start with HTTP long-polling (Socket.IO’s default, SockJS, and many home-grown fallbacks) make several independent HTTP requests that must all land on the node holding the session; round robin scatters them. Second, applications that keep per-user state in process memory — a presence map, a resume buffer — expect a reconnecting client to return to the same node, and round robin sends it elsewhere.

ip_hash fixes the scattering by hashing the client address (the first three octets for IPv4, the whole address for IPv6) and mapping it to a server. Its weaknesses come from what it hashes. Every user behind one corporate NAT or carrier-grade NAT shares an address, so a single office can land on one node. And the mapping is modulo the server list: add or remove a node and most keys move, so a scale-out event reshuffles a large share of reconnecting clients.

ip_hash is sticky, but it hashes the wrong thing for most real deployments.

Resolution #

Use hash … consistent on a per-client key when you can, and ip_hash only when the client has no stable identifier. The consistent variant uses ketama hashing, so adding a node moves roughly 1/N of keys instead of most of them. A good key is a session or client ID the client sends on every request — a query parameter or cookie — which is also NAT-proof.

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

# Prefer a stable client id (query ?cid= or cookie), fall back to the address.
map $arg_cid $sticky_key {
    ""      $cookie_rt_cid;
    default $arg_cid;
}
map $sticky_key $sticky_final {
    ""      $remote_addr;      # last resort: behaves like ip_hash, but per full address
    default $sticky_key;
}

upstream realtime {
    hash $sticky_final consistent;   # ketama: ~1/N of clients move on rescale
    server 10.0.1.11:8080 max_fails=3 fail_timeout=10s;
    server 10.0.1.12:8080 max_fails=3 fail_timeout=10s;
    server 10.0.1.13:8080 max_fails=3 fail_timeout=10s;
    keepalive 32;                    # pooled connections for the polling requests
}

server {
    listen 443 ssl;
    server_name rt.example.com;

    location /socket/ {
        proxy_pass http://realtime;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_read_timeout 3600s;
        proxy_send_timeout 3600s;
    }
}

If you must use ip_hash, make sure nginx sees the real client address. Behind another load balancer, $remote_addr is the balancer’s address and every client hashes to the same node; configure set_real_ip_from with the balancer’s range and real_ip_header X-Forwarded-For first, or better, switch to the header-based key above. The upgrade headers and timeouts in the location block are explained in configuring nginx for WebSocket upgrades.

The reason consistent hashing matters becomes obvious when you simulate a scale-out: with modulo hashing, going from three to four nodes remaps three quarters of keys; with a ketama ring, about a quarter.

Every remapped client that reconnects lands on a node without its in-memory state — consistent hashing keeps that number small.

Stickiness is a mitigation, not a design. State that must survive a node failure — presence, message history, session resume buffers — belongs in a shared store such as Redis, as described in scaling WebSocket broadcast with Redis pub/sub. With shared state, stickiness only protects the polling handshake and becomes far less critical.

Verification #

Confirm stickiness from the backend’s point of view by logging which node served each request of one client. Each backend can add a response header with its own identity:

# Every request with the same cid must report the same upstream.
for i in 1 2 3 4 5; do
  curl -s -o /dev/null -D - "https://rt.example.com/socket/?EIO=4&transport=polling&cid=user-42" \
    | grep -i x-upstream
done
# Add to the nginx location to expose it during testing:
#   add_header X-Upstream $upstream_addr always;

Then verify the distribution is not skewed. Count established connections per backend (ss -tn state established '( sport = :8080 )' | wc -l on each node). With ip_hash behind a large NAT, one node can hold several times its share; with the consistent client-ID hash the counts should be within a few percent of each other.

Consistent routing keeps every step of the handshake on the node that issued the session.

Operational checklist #

The upstream uses hash <client key> consistent, or ip_hash only when no client key exists.
The client sends a stable identifier on every request (query parameter or cookie).
nginx sees real client addresses (real_ip_header) if any address-based hashing is used.
Per-backend connection counts are graphed and alerted on for skew.
Adding a node is tested and the fraction of reconnecting clients is measured.
State needed after a node failure lives in a shared store, not only in process memory.

FAQ #

Do raw WebSockets need sticky sessions at all? #

A single WebSocket connection does not: after the upgrade it is one TCP connection pinned to one backend. You need stickiness only for multi-request handshakes (long-polling fallbacks) or when a reconnecting client must return to a node holding its in-memory state.

Why is one backend getting far more connections with ip_hash? #

ip_hash uses the first three octets of an IPv4 address, so every client in a /24 — and every user behind one NAT or proxy — maps to the same server. Large offices, universities and mobile carriers produce big buckets. Hash on a per-client identifier instead.

The sticky directive is part of the commercial nginx Plus. In open-source nginx, achieve the same effect with hash $cookie_name consistent, which routes on a cookie your application sets.

What happens to hashed clients when a backend fails? #

With max_fails, nginx marks the server unavailable and rehashes affected keys to the remaining servers; clients on the failed node reconnect and land elsewhere. With consistent hashing only that node’s clients move — everyone else stays put.

HAProxy WebSocket Load Balancing Configuration — stick tables and balance source in HAProxy.
Kubernetes ingress-nginx WebSocket Affinity — the same routing expressed as ingress annotations.
Configuring AWS ALB for WebSocket Sticky Sessions — cookie-based stickiness on a managed balancer.
Rebalancing WebSocket Connections After Scale-Out — moving load onto new nodes deliberately.

Back to Load Balancer Sticky Sessions.