Tracing messages across WebSocket and Redis #

A user types a message in a chat room, and a colleague in the same room sees it three seconds later. Your traces show the HTTP request that loaded the page, and they show the WebSocket upgrade from twenty minutes ago, but the journey of that one message — client to node A, node A to Redis, Redis to node B, node B to the colleague — appears nowhere. Distributed tracing works for HTTP because every hop forwards a traceparent header automatically. WebSocket frames and pub/sub messages have no headers, so the context is dropped at the first hop unless you carry it yourself. This page shows how to put W3C trace context into your message envelope and restore it on every hop.

Root cause #

OpenTelemetry’s automatic instrumentation propagates context through protocols it understands: HTTP headers, gRPC metadata, some message queues. A WebSocket frame is opaque bytes after the handshake, so the instrumentation can see the upgrade request but not the messages that follow; the upgrade span ends, and every later message handler starts with no active context — a new root trace, or no trace at all. Redis pub/sub is the same: PUBLISH channel payload carries only a payload, and the subscriber’s callback runs outside any trace.

The result is that the most latency-sensitive path in your system — publish-to-deliver across nodes — is invisible. The fix is manual propagation: inject the current context into a field of the message when sending, extract it when receiving, and start the handler’s span as a child (or link) of the extracted context.

One message, one trace A message carries a traceparent from the sender to node A, through Redis in its envelope, to node B and out to the receiver, with a span at each hop joined into one trace. One message, one trace Sender Node A Redis Node B Receiver frame + traceparent span: ws.receive PUBLISH envelope + ctx message + ctx span: pubsub.deliver span: ws.send Every arrow that crosses a process boundary carries the context inside the message
The trace survives because the envelope carries it, not because a transport does.

Resolution #

Add a tc (trace context) field to your message envelope and use the OpenTelemetry propagation API to inject and extract it. The propagator writes standard traceparent and tracestate keys into any object you give it, so the same code works for WebSocket frames and Redis payloads.

import { context, propagation, trace, SpanKind, ROOT_CONTEXT } from '@opentelemetry/api';
import type { WebSocket } from 'ws';
import type { RedisClientType } from 'redis';

const tracer = trace.getTracer('realtime');

interface Envelope { type: string; room: string; data: unknown; tc?: Record<string, string> }

// Write the active context into the envelope's carrier.
function inject(env: Envelope): Envelope {
  const carrier: Record<string, string> = {};
  propagation.inject(context.active(), carrier);   // adds traceparent (+ tracestate)
  return { ...env, tc: carrier };
}

// Rebuild a context from a received envelope (or start fresh if absent).
function extract(env: Envelope) {
  return env.tc ? propagation.extract(ROOT_CONTEXT, env.tc) : ROOT_CONTEXT;
}

// 1. Inbound WebSocket frame on node A.
export function onClientFrame(raw: string, pub: RedisClientType) {
  const env = JSON.parse(raw) as Envelope;
  const parent = extract(env);                    // browser may have sent a traceparent
  tracer.startActiveSpan('ws.receive', { kind: SpanKind.SERVER, attributes: {
    'messaging.system': 'websocket', 'messaging.destination.name': env.room, 'ws.message.type': env.type,
  } }, parent, async (span) => {
    try {
      // 2. Publish with the ws.receive span as the new parent.
      await tracer.startActiveSpan('pubsub.publish', { kind: SpanKind.PRODUCER }, async (pubSpan) => {
        await pub.publish(`room:${env.room}`, JSON.stringify(inject(env)));
        pubSpan.end();
      });
    } finally {
      span.end();
    }
  });
}

// 3. Subscriber callback on node B: continue the same trace, then fan out.
export function onPubSubMessage(payload: string, localSockets: Set<WebSocket>) {
  const env = JSON.parse(payload) as Envelope;
  tracer.startActiveSpan('pubsub.deliver', {
    kind: SpanKind.CONSUMER,
    attributes: { 'messaging.system': 'redis', 'realtime.fanout.local_count': localSockets.size },
  }, extract(env), (span) => {
    const out = JSON.stringify({ ...env, tc: undefined });  // don't leak context to clients by default
    for (const ws of localSockets) ws.send(out);
    span.end();
  });
}

Two decisions deserve care. First, strip the context before sending to end users unless you want browser-side spans in the same trace; the traceparent reveals nothing sensitive, but it is noise for most clients. Second, a message fanned out to 10,000 sockets should produce one delivery span per node with a count attribute, not 10,000 send spans; per-socket spans multiply trace volume by your room size. The broader span layout for connections is covered in instrumenting WebSockets with OpenTelemetry.

Browsers can join the trace too. If your frontend runs the OpenTelemetry web SDK, inject the active context into outbound frames the same way, and the trace starts at the user’s click rather than at your server.

Where the three seconds went Latency budget per path, split into client → node A, node A → node B, node B → client. Where the three seconds went per-hop span durations from one trace, ms client → node A node A → node B node B → client same node 26 ms cross node, Redis 29 ms 2.9k ms cross node, slow sub 2.9k ms
With context carried through Redis, the slow hop is obvious: the subscriber on node B was blocked, not the network.

Sampling and volume #

Real-time systems produce far more messages than HTTP systems produce requests, so tracing every message is rarely affordable. Use head sampling at a low rate for message traces — 0.1% is often enough to see latency distributions — and let the sampling decision propagate: because the sampled flag travels in traceparent, every downstream hop honours the upstream decision and traces stay complete rather than fragmented. For debugging a specific room or user, add an attribute-based rule in your collector’s tail sampler to keep 100% of traces matching that identifier for a limited time.

Keep the connection-level span short. A span open for the entire life of a socket is awkward in every tracing UI and is often dropped by exporters with maximum span durations. Record the upgrade as its own span, and link message spans to it with a span link carrying the connection id if you want to navigate between them.

Span design for message tracing The four span types for message tracing with their span kind, granularity and key attributes; delivery is one span per node rather than per socket. Span design for message tracing Kind Per Key attributes ws.upgrade SERVER connection conn_id, user_id ws.receive SERVER inbound message type, room pubsub.publish PRODUCER publish channel, bytes pubsub.deliver CONSUMER node, not socket local_count One delivery span per node keeps trace size independent of room size
Spans per hop, not per recipient.

Verification #

Send one message in a two-node test environment and find its trace in your backend (Jaeger, Tempo, Honeycomb). It should contain four spans — ws.receive, pubsub.publish, and one pubsub.deliver per node with subscribers — all under one trace id. If the delivery spans appear as separate root traces, the context is not surviving the Redis hop; log env.tc on the subscriber to confirm the field arrives.

# Quick check without a UI: the collector's debug exporter prints spans with trace ids.
docker logs otel-collector 2>&1 | grep -E 'Name +: (ws.receive|pubsub.publish|pubsub.deliver)' -A2 \
  | grep -E 'Name|Trace ID'

All three span names should list the same trace id. Then confirm sampling behaves: with a 0.1% ratio, a load test of 100,000 messages should produce roughly 100 complete traces, not 100 traces plus thousands of orphaned delivery spans.

Operational checklist #

FAQ #

Can OpenTelemetry auto-instrument WebSocket messages? #

Not in a standard way today: instrumentations cover the HTTP upgrade, but frames after the upgrade are opaque to them. Manual propagation through your envelope, as shown here, is the reliable approach.

Does this work with Redis Streams or Kafka instead of pub/sub? #

Yes. Streams entries and Kafka records can carry the context as a field or header; Kafka instrumentation even does it automatically. The extract-then-start-span pattern on the consumer is identical — see Redis Streams vs pub/sub for WebSocket fan-out.

How much overhead does the traceparent add? #

About 55 bytes per message for traceparent, plus any tracestate. Stripping it before sending to clients means the overhead exists only on internal hops, where it is negligible next to JSON framing.

A child keeps the trace as one tree, which is easiest to read for one-to-one and small fan-outs. For large fan-outs or batched deliveries, where one span processes messages from several traces, use span links to each originating context instead.

Back to WebSocket Observability & Monitoring.