Grafana Dashboards for WebSocket Health #

Most WebSocket dashboards are a wall of CPU and memory graphs with one connection count bolted on, and they answer none of the questions you have at 2 a.m. Is this a client problem or a server problem? Is one pod sick or all of them? Did connections drop, or did they never arrive? Are messages slow, or are they not being sent at all?

Six panels answer all of those, and every one of them is derived from metrics you should already be exporting. This page is the dashboard, the PromQL behind each panel, and the alert thresholds that make it actionable rather than decorative.

Root cause #

Request/response dashboards are built on a rate-and-duration model: requests per second, error percentage, latency percentiles. A WebSocket has one request and then hours of silence, so those panels flatline and tell you nothing.

The unit that matters instead is the connection and its lifecycle. A connection is opened, it lives, it exchanges messages, and it closes for a reason. That gives four families of signal — population, churn, health and throughput — and a dashboard that covers all four will localise almost any real-time incident in under a minute.

The other structural difference is cardinality. Per-connection metrics are a trap: 40,000 sockets with a label per session is 40,000 series per metric, which will take down your Prometheus faster than any outage. Every panel below aggregates by pod, close code or message type — dimensions with tens of values, not tens of thousands.

Four signal families, six panels A WebSocket dashboard covers four families of signal: how many connections exist, how fast they are being created and destroyed, how healthy the live ones are, and how much traffic they carry. Four signal families, six panels Population ws_connections_active by pod — how many sockets exist right now panels 1, 2 Churn opens and closes per second, close codes by reason panels 3, 4 Health heartbeat RTT, send-queue depth, dropped messages panel 5 Throughput messages in and out per second, by type panel 6 Localising an incident means finding which family moved first — the panel order is the triage order
Read the panels top to bottom during an incident: population, then churn, then health, then throughput.

Resolution #

The panels below assume the prom-client instrumentation from exporting WebSocket metrics to Prometheus. Each query is written to be safe on a fleet of any size — no per-session labels, no unbounded topk.

# 1. Active connections by pod — the population, and how evenly it is spread
sum by (pod) (ws_connections_active)

# 2. Fleet total against capacity — one number for "are we near the wall"
sum(ws_connections_active) / sum(ws_connection_capacity)

# 3. Connection churn: opens and closes per second, fleet-wide
sum(rate(ws_connections_opened_total[5m]))
sum(rate(ws_connections_closed_total[5m]))

# 4. Close reasons — the single most diagnostic panel on the dashboard
sum by (code) (rate(ws_connections_closed_total[5m]))

# 5a. Heartbeat round trip, p50/p95/p99 — client-side network health
histogram_quantile(0.99, sum by (le) (rate(ws_heartbeat_rtt_seconds_bucket[5m])))

# 5b. Send-queue depth p99 — backpressure, per pod
histogram_quantile(0.99, sum by (le, pod) (rate(ws_send_queue_bytes_bucket[5m])))

# 6. Message throughput by direction and type
sum by (direction, type) (rate(ws_messages_total[1m]))

Panel 4 is the one to build first. A healthy fleet shows a steady trickle of 1000 (clean closes), a background of 1006 (network events), and near-zero of everything else. Every incident shifts that mix in a characteristic way, which is what turns the panel into a diagnosis rather than a graph.

{
  "title": "Close reasons",
  "type": "timeseries",
  "targets": [{ "expr": "sum by (code) (rate(ws_connections_closed_total[5m]))", "legendFormat": "" }],
  "fieldConfig": { "defaults": { "custom": { "stacking": { "mode": "normal" } }, "unit": "cps" } },
  "options": { "legend": { "displayMode": "table", "calcs": ["mean", "max"] } }
}
Reading the close-code panel during an incident A triage table mapping each characteristic movement in the close-code panel to its most likely cause and the panel to check next. Reading the close-code panel during an incident What moved Most likely cause Next panel 1006 spike, one pod that pod is sick pod restarted or wedged queue depth by pod 1006 spike, all pods network or proxy LB idle timeout, ISP event heartbeat RTT 1001 spike deliberate drain deploy in progress opens per second 1013 appearing capacity shedding slow consumers or overload send-queue depth 1008 appearing policy rejections client release regression messages by type Opens up, active flat reconnect loop clients failing right after open close codes again The last row is the one people miss — a healthy connection count can hide a fleet churning through reconnects
Six patterns cover most real-time incidents, and each one names the next panel to open.

The final row deserves a dedicated panel of its own: sum(rate(ws_connections_opened_total[5m])) plotted next to sum(ws_connections_active). A fleet where opens are high and the population is flat is a fleet in a reconnect loop, and no other panel makes that visible.

Verification #

A dashboard is only trustworthy once you have watched it during a controlled failure.

# Force a drain on one pod and watch panels 1, 3 and 4 respond
kubectl delete pod ws-node-3 --grace-period=30

# Generate slow consumers and confirm the queue-depth panel moves before memory does
node scripts/slow-client.js --count 50 --read false

# Confirm cardinality has not exploded after a week of traffic
curl -s localhost:9090/api/v1/status/tsdb | jq '.data.seriesCountByMetricName[:5]'

The cardinality check is not optional. Run it after every metric change: if ws_messages_total has more than a few hundred series, something is labelling by user, room or session, and that will become an outage of its own.

How long a dead socket survives, by heartbeat interval Detection latency for heartbeat intervals 10, 20, 30, 60 seconds with a 10 second pong timeout: worst case is always interval plus timeout. How long a dead socket survives, by heartbeat interval pong timeout 10s — blue = best case, red = worst case (interval + timeout) 0s 20s 40s 60s 80s 10s 20s 30s 60s
Why the heartbeat panel needs its interval annotated: the same RTT histogram means very different detection latency depending on the interval configured, and the dashboard should say which.

Operational checklist #

FAQ #

What alert thresholds actually work? #

Ratios and rates of change, not absolutes. A useful starting set: abnormal closes above 20% of all closes over five minutes; active connections dropping more than 30% in two minutes; p99 heartbeat RTT above two seconds; p99 send-queue depth above your high-water mark for five minutes. Absolute connection-count alerts break the first time traffic grows, and everyone learns to ignore them.

Why is my connection count different from my load balancer’s? #

Usually because they count different things. Your gauge counts sockets the application believes are open, which includes half-open connections whose peers have gone away, while the load balancer counts its own established connections. A persistent gap between the two is the signature of the leak described in fixing WebSocket CLOSE_WAIT accumulation — which is exactly why exporting both is worth the effort.

Should I track per-room or per-user metrics? #

Not in Prometheus. Aggregate dimensions with bounded cardinality belong in metrics; anything keyed by user, room or session belongs in logs or traces, where high cardinality is expected and priced accordingly. If you need per-room visibility, sample: export metrics for a small allow-list of high-value rooms rather than all of them.

How do I chart message latency end to end? #

Put a producer timestamp in the envelope and have the client report the delta back on a low-frequency schedule — one sample per client per minute is plenty. Server-side histograms alone cannot see the network leg, which is usually where the latency is, and the message framing and serialization envelope already carries the timestamp field this needs.

Does this work with OpenTelemetry instead of Prometheus? #

Yes. The metric names and semantics are identical; only the export path changes, and the OTel collector can emit a Prometheus endpoint so the same queries work unchanged. Where OTel adds real value is connecting a connection’s spans to the same trace as the HTTP request that authorised it, which is covered in instrumenting WebSockets with OpenTelemetry.

Back to WebSocket Observability & Monitoring