Alerting on WebSocket connection drops #

Your on-call engineer is paged at 2 a.m. because the connected-clients gauge dropped by 8%. It was a deploy. Two weeks later, a CDN change silently caps idle sockets at 100 seconds, a quarter of all connections reconnect every two minutes, the gauge barely moves, and nobody is paged at all. Connection counts are the obvious thing to alert on and the worst signal to use on their own: they move for benign reasons and stay flat through serious failures, because reconnects hide drops. Useful WebSocket alerts measure churn and abnormal closure, normalised by the size of the fleet, and they know when a deploy is happening.

Root cause #

A connection gauge is a stock, and drops are a flow. When clients reconnect quickly — which good clients do — every drop is followed by a new connection within seconds, and the stock barely changes even if the flow is enormous. A fleet where every client reconnects once a minute can show a perfectly flat gauge while users see a spinner every minute and your handshake path burns CPU. Conversely, the gauge falls sharply and legitimately at every deploy, at the end of the working day, and whenever a large customer closes their dashboards.

The signals that do correspond to user impact are rates: abnormal closes per second relative to open connections, new connections per second relative to open connections (churn), and handshake failures relative to handshake attempts. Each is a ratio, so it holds its meaning as the fleet grows.

Why the gauge hides a reconnect loop An edge proxy cuts an idle connection, the gauge decrements, the client reconnects a second later and the gauge increments, so the net change is zero even though the user was disconnected. Why the gauge hides a reconnect loop Client Edge Server Gauge idle cut at 100 s −1 connection reconnect after 1 s +1 connection net change: 0 close 1006 + new open The close counter and the open counter both moved — that is where the alert belongs
Stocks hide flows. Alert on the flow.

Resolution #

Export three counters and one gauge from the server — the same families described in exporting WebSocket metrics to Prometheus — and build alerts on ratios over windows long enough to smooth normal noise. Suppress the drop-based alerts while a deploy is in progress by joining against a deploy marker metric, rather than silencing the whole alert group by hand.

groups:
  - name: websocket-connection-health
    rules:
      # Share of open connections closing abnormally per minute (1006, 1011, 1013 …).
      - record: ws:abnormal_close_ratio:rate5m
        expr: |
          sum(rate(ws_connections_closed_total{code!~"1000|1001"}[5m]))
          /
          clamp_min(sum(ws_connections_open), 1)

      # Churn: new connections per open connection per minute. Healthy fleets sit far below 0.05.
      - record: ws:churn_ratio:rate5m
        expr: |
          sum(rate(ws_connections_opened_total[5m])) * 60
          /
          clamp_min(sum(ws_connections_open), 1)

      - alert: WebSocketAbnormalCloseRateHigh
        expr: |
          ws:abnormal_close_ratio:rate5m * 60 > 0.02
          unless on() max(deploy_in_progress) == 1
        for: 10m
        labels: { severity: page }
        annotations:
          summary: "More than 2% of connections per minute are closing abnormally"
          runbook: "https://runbooks.example.com/ws/abnormal-close"

      - alert: WebSocketReconnectChurnHigh
        expr: ws:churn_ratio:rate5m > 0.1 unless on() max(deploy_in_progress) == 1
        for: 15m
        labels: { severity: ticket }
        annotations:
          summary: "Clients are reconnecting unusually often — check idle timeouts on the path"

      - alert: WebSocketHandshakeFailuresHigh
        expr: |
          sum(rate(ws_upgrades_total{result!="accepted"}[5m]))
          / clamp_min(sum(rate(ws_upgrades_total[5m])), 1) > 0.05
        for: 5m
        labels: { severity: page }

      # Gauge cliff that is NOT a deploy: a sudden fleet-wide loss.
      - alert: WebSocketConnectionsCliff
        expr: |
          sum(ws_connections_open) < 0.6 * sum(ws_connections_open offset 15m)
          unless on() max(deploy_in_progress) == 1
        for: 3m
        labels: { severity: page }

The thresholds are starting points. Measure your own baselines for a week first: record the two ratios, look at their daily shape, and set the alert above the normal peak with margin. A consumer app on mobile networks has far more natural churn than an internal dashboard on office Wi-Fi.

deploy_in_progress can come from your deploy tool pushing a gauge to a Pushgateway, or from a recording rule over kube_deployment_status_observed_generation versus kube_deployment_metadata_generation. Suppress only the drop-based alerts during deploys — handshake failures during a rollout are exactly what you want to hear about, since they mean the new version is rejecting clients. Graceful deploys that close with 1001 do not count as abnormal anyway, which is one more reason to shut down gracefully.

Ratios in four situations In healthy operation both ratios are low; a graceful deploy raises churn but not abnormal closes; an idle cut and a bad release raise both. Ratios in four situations percent of open connections per minute abnormal closes %/min churn %/min 0% 10% 20% 30% Healthy 9% Deploy (1001) 9% CDN idle cut 14% 18% Bad release
A graceful deploy moves churn only; real incidents move both ratios — which is why the page fires on abnormal closes.

Edge cases #

Small fleets and quiet hours. Ratios are noisy when the denominator is small: at 3 a.m. with 200 connections, four abnormal closes in a minute is 2%. Add a minimum-volume condition, such as and sum(ws_connections_open) > 1000, or use a longer window for low-traffic periods, so the page reflects user impact rather than arithmetic.

Multi-region fleets. A fleet-wide ratio can hide a regional outage: one region at 30% abnormal closes and three healthy regions average out below the threshold. Compute the ratios by (region) and alert per region, keeping the fleet-wide rule as a backstop.

Client-caused closes. Browsers closing tabs produce 1001 from the client side, which should not count against you, and some mobile SDKs close with 1000 when backgrounded. Make sure the code label records who initiated the close if your server can tell, so client-initiated closes can be excluded from the abnormal ratio.

Verification #

Test alerts against reality before trusting them. Replay a past incident with promtool test rules, feeding the series you recorded during the event, and assert that the alert fires within the expected time and stays silent during a deploy window:

# ws_alerts_test.yml — run with: promtool test rules ws_alerts_test.yml
rule_files: [ws_alerts.yml]
evaluation_interval: 1m
tests:
  - interval: 1m
    input_series:
      - series: 'ws_connections_open'
        values: '10000x30'
      - series: 'ws_connections_closed_total{code="1006"}'
        values: '0+300x30'          # 300 abnormal closes per minute = 3%
      - series: 'deploy_in_progress'
        values: '0x30'
    alert_rule_test:
      - eval_time: 20m
        alertname: WebSocketAbnormalCloseRateHigh
        exp_alerts: [{ exp_labels: { severity: page } }]

Run a second test with deploy_in_progress set to 1 and assert no alert. Then do a game day: cap idle timeouts on a staging proxy to 30 seconds and confirm the churn alert opens a ticket.

A CDN idle cut, as the alerts see it A CDN configuration change raises the churn ratio within two minutes and abnormal closes within five, the page fires at fifteen minutes and the change is rolled back at thirty. A CDN idle cut, as the alerts see it users see reconnects CDN config change (0 min) churn ratio climbs (2 min) abnormal closes 9%/min (5 min) page fires (for: 10m) (15 min) churn ticket opens (17 min) config rolled back (30 min) The gauge stayed within 3% of normal for the whole incident
Ratio alerts catch the incident the gauge never showed.

Operational checklist #

FAQ #

Why not just alert when connected users drop? #

Because reconnecting clients keep the number flat during real incidents, and deploys and daily cycles move it during non-incidents. Keep a gauge-cliff alert for catastrophic losses, but rely on close and churn ratios for everything else.

Which close codes count as abnormal? #

Everything except 1000 (normal) and 1001 (going away) is worth counting, with 1006 usually dominating. Break the ratio down by code on the dashboard: a rise in 1006 points at the network or proxies, 1011 at server errors, 1013 at load shedding, and your 4xxx codes at auth.

How do I avoid alert noise from mobile clients? #

Normalise by open connections, use windows of five minutes or more, and require the condition to hold for ten minutes. If mobile churn is inherently high, split the ratio by a client-type label and alert on each population against its own baseline.

Should client-side telemetry feed these alerts? #

It is a valuable second source — clients can report failures the server never sees, like handshakes that never reached you. Use it for dashboards and for confirming an incident, but keep paging on server metrics, which are more complete and harder to spoof.

Back to WebSocket Observability & Monitoring.