Kubernetes ingress-nginx WebSocket affinity #

Your WebSocket service works perfectly through kubectl port-forward and falls apart behind the cluster’s ingress: connections drop every 60 seconds, polling-transport clients bounce between pods and lose their sessions, and a rolling deploy disconnects everyone at once rather than gradually. The ingress-nginx controller supports WebSockets out of the box — it forwards the Upgrade header without extra configuration — but its defaults are tuned for short HTTP requests. Affinity, timeouts and connection handling are all controlled by annotations on the Ingress object, and each one has a trap. This page walks through a working configuration and explains every setting.

Root cause #

ingress-nginx renders your Ingress resources into an nginx configuration, and three of its defaults hurt long-lived connections. The proxy read and send timeouts default to 60 seconds, so an idle socket is closed by the controller — the same problem covered generically in tuning WebSocket idle timeouts across proxies. Load balancing across pod endpoints defaults to round robin, so multi-request handshakes scatter across pods. And when the controller itself reloads — which happens whenever any ingress in the cluster changes — old nginx worker processes keep serving existing connections only until worker-shutdown-timeout expires, after which every WebSocket they hold is cut.

Most WebSocket settings belong on the Ingress object; a few are controller-wide.

A common mistake is setting sessionAffinity: ClientIP on the Service and expecting it to work. ingress-nginx bypasses kube-proxy and sends traffic directly to pod endpoints, so Service-level affinity is never consulted. Affinity has to be configured on the ingress.

Resolution #

The Ingress below enables cookie affinity, raises the proxy timeouts, and disables buffering for the WebSocket path. Cookie affinity is the most reliable option because the controller issues the cookie itself and the browser returns it on every request, including the upgrade.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: realtime
  annotations:
    # Idle tolerance for upgraded connections (seconds). Keep above the app heartbeat.
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
    # Sticky routing: the controller sets and honours this cookie.
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/affinity-mode: "persistent"    # keep stickiness when pods are added
    nginx.ingress.kubernetes.io/session-cookie-name: "rt_route"
    nginx.ingress.kubernetes.io/session-cookie-max-age: "86400"
    nginx.ingress.kubernetes.io/session-cookie-samesite: "Lax"
    nginx.ingress.kubernetes.io/session-cookie-secure: "true"
    # Streaming responses (polling fallback) must not be buffered.
    nginx.ingress.kubernetes.io/proxy-buffering: "off"
spec:
  ingressClassName: nginx
  tls:
    - hosts: [rt.example.com]
      secretName: rt-example-tls
  rules:
    - host: rt.example.com
      http:
        paths:
          - path: /ws
            pathType: Prefix
            backend:
              service:
                name: realtime
                port:
                  number: 8080

For clients that cannot hold cookies — native apps, server-to-server consumers — use hash-based routing instead: nginx.ingress.kubernetes.io/upstream-hash-by: "$arg_cid" routes on a query parameter with consistent hashing, the same idea explained in WebSocket sticky sessions with nginx ip_hash. Do not combine it with cookie affinity on the same ingress; pick one.

The controller-wide settings go in the controller’s ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
data:
  # How long old workers keep existing connections after a config reload.
  worker-shutdown-timeout: "600s"
  # Reuse upstream connections for the HTTP parts of the handshake.
  upstream-keepalive-connections: "64"

worker-shutdown-timeout is the setting most teams discover last. Every ingress change anywhere in the cluster triggers a reload; with a short timeout, an unrelated team’s deploy disconnects your users. Raising it lets old workers keep serving existing WebSockets while new workers take new connections, at the cost of extra memory during the overlap.

The shutdown timeout decides whether a cluster-wide reload is invisible or a mass disconnect.

Edge cases #

Cookie paths and multiple ingresses. The affinity cookie is scoped to the path of the ingress rule by default. If your WebSocket endpoint lives under /ws and your polling fallback under /socket.io, defined in two different ingress objects, each gets its own cookie and the two can route to different pods. Put every path that belongs to one session in the same ingress, or set session-cookie-path: "/" explicitly.

SameSite and cross-site embedding. When the real-time endpoint is on a different registrable domain from the page — an embedded widget, for example — a Lax cookie is not sent on the cross-site WebSocket handshake, and affinity silently stops working. Either serve the endpoint from the same site or set session-cookie-samesite: "None" with Secure, and review the implications for cross-origin WebSocket connections.

TLS passthrough bypasses all of this. With ssl-passthrough enabled, the controller routes raw TLS by SNI and never sees HTTP, so annotations for affinity and timeouts do nothing. Passthrough is occasionally used for mutual TLS terminated at the pod; if you use it, stickiness and idle timeouts become the pod’s and the cloud load balancer’s problem.

Connection limits per worker. Each controller worker accepts up to max-worker-connections (default 16384), and an upgraded socket consumes two — one downstream, one upstream. A single controller replica with the default worker count tops out well below what the pods behind it can hold. Size the controller Deployment by concurrent sockets, not by request rate.

Verification #

Inspect the configuration the controller actually rendered, because a typo in an annotation is silently ignored:

POD=$(kubectl -n ingress-nginx get pod -l app.kubernetes.io/component=controller -o name | head -1)
# The rendered server block for your host: look for proxy_read_timeout and the affinity balancer.
kubectl -n ingress-nginx exec "$POD" -- cat /etc/nginx/nginx.conf \
  | sed -n '/server_name rt.example.com/,/^\t}/p' | grep -E 'proxy_(read|send)_timeout|upgrade'
# Confirm the cookie is issued on the first request.
curl -sI https://rt.example.com/ws/health | grep -i set-cookie

Then test stickiness end to end: repeat a request with the returned cookie and confirm the same pod answers each time (log the pod name in a response header during testing). Finally, trigger a harmless reload by adding an annotation to an unrelated ingress, and watch your connection gauge — it should not dip.

Pick persistent cookies for browsers, a hash key for everything else.

Operational checklist #

proxy-read-timeout and proxy-send-timeout exceed the application heartbeat interval.
Affinity is set on the Ingress (cookie or upstream-hash-by), not on the Service.
worker-shutdown-timeout in the controller ConfigMap is long enough to cover typical connection lifetimes.
The rendered nginx.conf is checked after every annotation change.
The cloud load balancer in front of the controller has an idle timeout above the heartbeat interval.
Pods drain gracefully on SIGTERM so rolling deploys stagger reconnects.

FAQ #

Do I need an annotation to enable WebSockets on ingress-nginx? #

No. The controller passes the Upgrade and Connection headers through by default. What you do need are longer proxy timeouts, because the 60-second defaults close idle sockets.

Why do connections drop whenever someone deploys an unrelated app? #

Any ingress change reloads the controller, and old nginx workers are shut down after worker-shutdown-timeout. Connections they still hold are cut at that point. Raise the timeout in the controller ConfigMap.

Does affinity keep a client on the same pod across deploys? #

No. When the pod is replaced its connections end regardless of affinity, and the cookie points at an endpoint that no longer exists, so the controller picks a new one. Affinity only helps while the target pod is alive; plan for graceful draining instead.

Is the Gateway API different? #

The Gateway API expresses timeouts and session persistence in its own resources, and support varies by implementation. The concepts carry over directly: long idle timeouts on the route, affinity by cookie or header, and graceful handling of data-plane reloads.

WebSocket Sticky Sessions with nginx ip_hash — the underlying nginx balancing options.
Envoy WebSocket Proxy Configuration — the equivalent for Envoy-based gateways.
Autoscaling WebSockets on Kubernetes with KEDA — scaling the pods behind this ingress.
WebSocket Readiness and Liveness Probes — keeping the endpoint list honest.

Back to Load Balancer Sticky Sessions.