Envoy WebSocket proxy configuration #

Your WebSocket handshake through Envoy returns 404 or 426, or it succeeds and the connection dies after exactly 15 seconds, or after an hour, depending on which timeout you hit first. Envoy is the data plane under Istio, Contour, Emissary and many API gateways, and unlike nginx it does not proxy WebSocket upgrades by default. Upgrades must be enabled explicitly on the HTTP connection manager, and three separate timeouts — the route timeout, the stream idle timeout and the connection idle timeout — apply to long-lived streams in ways that are easy to misread. This page gives a working static configuration and explains each field.

Root cause #

Envoy treats an HTTP/1.1 upgrade as a special kind of stream. Unless the connection manager lists websocket in its upgrade_configs, Envoy strips the Upgrade header and forwards a normal request, which your server rejects or answers with a plain HTTP response. Once upgrades are enabled, the upgraded connection is still a stream inside Envoy’s model, so stream-level limits apply to it.

The route timeout (default 15 seconds) is the one that surprises people. It bounds the time until the upstream response is complete, and for a WebSocket the “response” never completes, so the connection is killed 15 seconds after the upgrade. The stream_idle_timeout on the connection manager (default five minutes) closes streams with no activity, and a cluster’s idle_timeout under common_http_protocol_options can recycle upstream connections. Each needs a deliberate value.

Three layers each have a timer that applies to upgraded streams.

Resolution #

The static configuration below enables upgrades, disables the route response timeout for the WebSocket route, sets a generous but finite stream idle timeout, and uses ring-hash load balancing keyed on a client identifier header so a reconnecting client returns to the same upstream.

static_resources:
  listeners:
    - name: realtime_listener
      address: { socket_address: { address: 0.0.0.0, port_value: 8443 } }
      filter_chains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                stat_prefix: realtime
                upgrade_configs:
                  - upgrade_type: websocket          # without this, Upgrade is stripped
                stream_idle_timeout: 3600s           # idle streams (no frames either way) closed after 1 h
                route_config:
                  virtual_hosts:
                    - name: realtime
                      domains: ["rt.example.com"]
                      routes:
                        - match: { prefix: "/ws" }
                          route:
                            cluster: realtime_backend
                            timeout: 0s              # no response deadline for upgraded streams
                            hash_policy:
                              - header: { header_name: x-client-id }  # affinity key
                              - connection_properties: { source_ip: true }
                http_filters:
                  - name: envoy.filters.http.router
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
  clusters:
    - name: realtime_backend
      type: STRICT_DNS
      lb_policy: RING_HASH                          # consistent hashing on hash_policy
      ring_hash_lb_config: { minimum_ring_size: 1024 }
      load_assignment:
        cluster_name: realtime_backend
        endpoints:
          - lb_endpoints:
              - endpoint: { address: { socket_address: { address: realtime, port_value: 8080 } } }

The two entries under hash_policy are evaluated in order: if the x-client-id header is present it is used, and if not, the source IP becomes the key. Browsers cannot set custom headers on the WebSocket handshake, so for browser clients either put the identifier in a cookie and hash on cookie, or accept source-IP hashing with the NAT caveats described in WebSocket sticky sessions with nginx ip_hash.

Keep the application heartbeat running even with a one-hour stream_idle_timeout: other hops in the path are usually shorter, and the ping keeps each of them reset. The interval arithmetic is in implementing WebSocket ping-pong in Node.js.

One timeout to disable, one to raise, and two to understand.

Edge cases #

Retries and upgrades do not mix. A route-level retry_policy is harmless for the handshake itself, but Envoy cannot retry an upgraded stream once frames have flowed, and a retry on reset during the handshake can land the client on a different upstream than the hash chose. Keep retries off the WebSocket route, or restrict them to connect-failure so only a refused TCP connection is retried.

Circuit breakers count upgraded streams. Each open WebSocket occupies one request slot against the cluster’s max_requests circuit breaker (default 1024) and, for HTTP/1.1 upstreams, one connection against max_connections. A fleet with ten thousand sockets per Envoy instance will start seeing 503 with the UO (upstream overflow) response flag on new handshakes long before any server is busy. Raise both thresholds for the WebSocket cluster to comfortably above your expected concurrent connection count per proxy.

Per-connection buffer limits apply in both directions. per_connection_buffer_limit_bytes (default 1 MiB) caps how much Envoy buffers for a slow reader before applying backpressure to the other side. That is the behaviour you want — it stops a slow client from growing Envoy’s memory — but it also means a burst larger than the limit will stall the upstream’s writes, which then shows up as bufferedAmount growth in your server, the scenario covered in handling WebSocket bufferedAmount backpressure.

Header-based hashing needs a header the client can send. Browsers cannot attach arbitrary headers to new WebSocket(), so an x-client-id hash only works for native and server clients. For browsers, hash on a cookie with hash_policy: [{ cookie: { name: rt_cid } }], which Envoy can also generate for you when the ttl field is set.

Verification #

Check the handshake first: a correct setup returns 101 Switching Protocols through Envoy. Then check that the connection survives past both the old route timeout and your intended idle bound:

# 1. Handshake through Envoy (expect 101).
curl -si --http1.1 -H 'Connection: Upgrade' -H 'Upgrade: websocket' \
  -H 'Sec-WebSocket-Version: 13' -H 'Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==' \
  https://rt.example.com/ws | head -1
# 2. Envoy's own counters: upgrades accepted, and streams closed by idle timeout.
curl -s localhost:9901/stats | grep -E 'realtime\.downstream_cx_upgrades_total|downstream_rq_idle_timeout'

A climbing downstream_rq_idle_timeout counter means streams are being closed by the idle timer — either the heartbeat is not running or the timeout is shorter than you think. For affinity, log the upstream host (%UPSTREAM_HOST% in the access log format) and confirm that repeated connections with the same x-client-id land on the same endpoint.

Envoy forwards the upgrade only because upgrade_configs lists websocket.

Operational checklist #

upgrade_configs includes websocket on the HTTP connection manager.
The WebSocket route sets timeout: 0s (or an explicit long value).
stream_idle_timeout is longer than the application heartbeat interval.
The cluster uses RING_HASH or MAGLEV with a hash policy when reconnect affinity matters.
Access logs include the upstream host so affinity can be verified.
Envoy stats for upgrades and idle timeouts are scraped and graphed.

FAQ #

Why does my WebSocket through Envoy close after 15 seconds? #

That is the default route timeout, which bounds how long Envoy waits for the upstream response to complete. An upgraded connection never completes, so it is reset at 15 seconds. Set timeout: 0s on the WebSocket route.

Does Istio need the same changes? #

Istio configures Envoy for you and enables WebSocket upgrades on HTTP routes, but you still need a VirtualService timeout suited to long connections and a DestinationRule for consistent-hash affinity. See Istio service mesh WebSocket configuration.

Can Envoy proxy WebSockets over HTTP/2? #

Yes. Envoy supports the extended CONNECT method from RFC 8441, so a client can open a WebSocket over an HTTP/2 connection. Enable allow_connect in the HTTP/2 protocol options; see the WebSocket handshake over HTTP/2 for the protocol details.

How do I drain WebSockets when Envoy itself restarts? #

Envoy’s hot restart hands listeners to the new process and lets the old one drain for --drain-time-s. Existing upgraded streams stay on the old process until they close or the parent shutdown time elapses, so set those flags long enough for your typical connection lifetime.

Kubernetes ingress-nginx WebSocket Affinity — the nginx-based ingress equivalent.
HAProxy WebSocket Load Balancing Configuration — tunnel timeouts and stick tables.
Tuning WebSocket Idle Timeouts Across Proxies — finding the shortest timer on the path.
Istio Service Mesh WebSocket Configuration — Envoy under a mesh control plane.

Back to Load Balancer Sticky Sessions.