Building a Kafka to WebSocket bridge #

Your order events, sensor readings and audit trail already flow through Kafka, and the product team wants them live in the browser. The first prototype creates a Kafka consumer per WebSocket connection, subscribed to the topic and filtering for the user’s data. It works for ten users. At five hundred, the Kafka cluster is rebalancing consumer groups constantly, brokers are serving the same partitions hundreds of times over, and connection churn causes minute-long delivery stalls. Kafka is an excellent source for real-time UIs, but it is not a fan-out system for browsers: its unit of consumption is a partition read by a consumer group member, not a per-user channel. A bridge sits between the two, consuming Kafka once per node and routing to sockets in memory.

Root cause #

Kafka organizes data into topics split into partitions; each consumer group divides the partitions among its members and tracks a committed offset per partition. That model is built for a modest number of long-lived consumers processing everything — not thousands of short-lived, selective ones.

Mapping one WebSocket to one consumer breaks it in three ways. Consumer group churn: every connect and disconnect triggers a group rebalance, pausing consumption for all members while partitions are reassigned. Redundant reads: each per-socket consumer (in its own group, to see all partitions) re-reads the entire topic from the brokers, multiplying broker egress by the number of users. Wasted filtering: each consumer discards almost everything it reads, because each user cares about a tiny slice.

Bridge architecture Kafka partitions keyed by account id are consumed once per bridge node, filtered through an in-memory map from account to local sockets, and delivered through WebSocket nodes to browsers that resume with their last applied offset. Bridge architecture Kafka topic (partitions) orders, keyed by account id; source of truth and replay log source Bridge consumers one consumer per bridge node, shared group or broadcast consume once In-memory routing account id → set of local sockets filter here WebSocket nodes hold connections; may be the same process as the bridge deliver Browsers resume with the last offset they applied clients Kafka is read once per node; per-user filtering happens in memory
Consume once, route in memory — never one consumer per socket.

Resolution #

Choose one of two layouts depending on whether WebSocket clients can be routed to the node that owns their partitions.

Broadcast layout (simpler). Every bridge node consumes every partition — each node uses its own consumer group, or assigns all partitions manually without group management — and routes events to its local sockets. Broker egress is nodes × topic throughput, which is fine for moderate volumes and a handful of nodes.

Partition-affinity layout (scales further). Bridge nodes share one consumer group, so each partition is consumed by exactly one node, and a load balancer routes each client to the node that owns its key’s partition (consistent hashing on the same key Kafka partitions by). Broker egress is 1 × topic throughput, but rebalances move clients between nodes.

The code below implements the broadcast layout with KafkaJS and manual offsets, since per-node groups with no commits keep it simple and let clients resume from Kafka offsets directly.

import { Kafka, logLevel } from 'kafkajs';
import type { WebSocket } from 'ws';

const kafka = new Kafka({ clientId: `ws-bridge-${process.env.HOSTNAME}`, brokers: ['kafka-0:9092', 'kafka-1:9092'], logLevel: logLevel.WARN });
// Own group per node: every node sees every partition; nothing is shared or rebalanced across nodes.
const consumer = kafka.consumer({ groupId: `ws-bridge-${process.env.HOSTNAME}`, sessionTimeout: 30_000 });

const socketsByAccount = new Map<string, Set<WebSocket>>();
const SKIP_IF_BUFFERED = 1_000_000;

await consumer.connect();
await consumer.subscribe({ topic: 'orders', fromBeginning: false });   // live tail; resume handled below

await consumer.run({
  autoCommit: false,                         // offsets are per-client resume tokens, not group state
  eachMessage: async ({ partition, message }) => {
    const account = message.key?.toString();
    if (!account) return;
    const targets = socketsByAccount.get(account);
    if (!targets) return;                    // nobody on this node cares: cheap drop
    // The (partition, offset) pair is a precise resume point for this account's stream.
    const frame = JSON.stringify({
      type: 'order.event', partition, offset: message.offset, data: JSON.parse(message.value!.toString()),
    });
    for (const ws of targets) {
      if (ws.readyState === ws.OPEN && ws.bufferedAmount < SKIP_IF_BUFFERED) ws.send(frame);
    }
  },
});

export function register(account: string, ws: WebSocket) {
  let set = socketsByAccount.get(account);
  if (!set) socketsByAccount.set(account, (set = new Set()));
  set.add(ws);
  ws.once('close', () => { set!.delete(ws); if (set!.size === 0) socketsByAccount.delete(account); });
}

For resume, a client reconnecting with { partition, offset } can be served its missed events by a short-lived reader that fetches from that offset for that partition, filters by key, and hands off to the live tail. Because an account’s events are all in one partition (Kafka partitions by key), a single offset is a complete resume token — the same idea as the sequence-based resume in resuming WebSocket sessions after reconnect. If the gap is older than the topic’s retention, fall back to a snapshot from the system of record.

When an event must reach users on many nodes, or topics carry more than the bridge nodes can each consume, put a fan-out bus between the bridge and the WebSocket tier: a small number of bridge consumers publish each event to a per-account channel on Redis or NATS, and WebSocket nodes subscribe only to accounts with local members — see NATS vs Redis for WebSocket fan-out.

Times the topic is read from Kafka A consumer per socket reads the topic once per user, up to ten thousand times; the broadcast layout reads it once per bridge node, four times; the partition-affinity layout reads it once. Times the topic is read from Kafka multiples of topic throughput served by brokers consumer per socket broadcast (4 nodes) partition affinity 0× 2.5k× 5.0k× 7.5k× 10k× 100 users 1,000 users 10,000 users
Per-socket consumers multiply broker load by the user count; a bridge reads the topic a constant number of times.

Edge cases #

Hot keys. Kafka partitions by key, so one very active account lands on one partition and one consumer thread. If a single key’s rate exceeds what the bridge can route, the bottleneck is the key, not the cluster — batch its events before sending, as in coalescing high-frequency WebSocket updates.

Consumer lag and live UIs. A bridge that falls behind delivers stale “live” data. Export consumer lag per partition and alert on it in seconds, not messages; a lagging bridge should shed non-critical traffic rather than deliver everything late.

Authorization. The bridge sees every event on the topic. Filtering by account key is also an authorization boundary: register sockets only under accounts the authenticated user may see, following per-channel authorization for WebSocket subscriptions.

Verification #

Watch the Kafka side and the WebSocket side together. On the Kafka side, the number of consumer groups reading the topic should equal the number of bridge nodes (broadcast layout) or one (affinity layout), and it must not change when users connect and disconnect:

kafka-consumer-groups.sh --bootstrap-server kafka-0:9092 --list | grep ws-bridge | wc -l
kafka-consumer-groups.sh --bootstrap-server kafka-0:9092 --describe --group ws-bridge-node-a   # lag per partition

On the WebSocket side, measure produce-to-browser latency by stamping events at production and comparing in the client. Then test resume: disconnect a client, produce events for its account, reconnect with the last offset, and confirm the missed events arrive exactly once.

Resume from a Kafka offset A reconnecting browser sends its last partition and offset; the bridge fetches partition three from the next offset, filters by the account key, delivers the missed events and then hands off to the live tail. Resume from a Kafka offset Browser Bridge node Kafka reconnect: partition 3, offset 88120 fetch p3 from 88121 (key filter) events 88121–88140 missed events for this account hand off to live tail Keying by account keeps every event for an account in one partition, so one offset is a complete resume point
Kafka's log is the replay buffer — the client just needs its offset.

Operational checklist #

FAQ #

Can browsers consume Kafka directly? #

No. Kafka’s protocol is a binary TCP protocol for trusted clients, and exposing brokers to browsers would be neither practical nor safe. A bridge service consumes Kafka and serves browsers over WebSockets or Server-Sent Events.

Should each WebSocket connection have its own Kafka consumer? #

No. It causes constant consumer group rebalances and multiplies broker reads by the number of users. Consume once per bridge node and route to sockets in memory.

How do I replay missed events after a reconnect? #

Send the partition and offset with each event; on reconnect, read from the next offset for that partition, filter by the client’s key, and then switch to the live stream. Kafka’s retention defines how far back that works.

Is Kafka too slow for real-time UIs? #

Kafka’s end-to-end latency is typically milliseconds to tens of milliseconds, depending on producer batching (linger.ms) and consumer polling. For live dashboards that is usually fine; tune producer batching down for the most latency-sensitive topics.

Back to Redis Pub/Sub Fan-Out.