Ordering WebSocket messages with sequence numbers #

A single WebSocket connection delivers messages in order — TCP guarantees it. So why do users see a chat reply appear above the message it answers, a task marked “done” before it was “in progress”, or a counter jump backwards? Because order on one connection says nothing about order across the system. Messages for one room are published from several API servers, pass through a pub/sub layer, fan out from several WebSocket nodes, and are resent after reconnects. Each hop is ordered on its own, and the whole is not. Sequence numbers — a monotonic counter per stream, assigned once, carried everywhere — are how you get a total order back, detect what went missing, and throw away duplicates.

Root cause #

Ordering breaks at the points where messages from different sources merge or where delivery is retried. Two API servers handling concurrent writes to the same room publish in whatever order their requests finish, and the pub/sub layer delivers them in the order it received them, which may not match the order the database committed them. A client that reconnects and resumes receives a replay followed by live messages, and the boundary between them can overlap or leave a hole. And an at-least-once delivery layer, as in at-least-once WebSocket delivery with acknowledgements, redelivers messages by design.

Timestamps do not fix any of this. Clocks on different servers disagree by milliseconds or more, two events can share a timestamp, and — crucially — a timestamp cannot tell you that a message is missing. A sequence number can: if the last one you applied was 41 and the next is 43, message 42 exists and you do not have it.

Where order is lost API one commits A and API two commits B shortly after, but B's publish arrives at the pub/sub layer first, so the client receives B before A; with sequence numbers assigned at commit, the client can put them back in order. Where order is lost API 1 API 2 Pub/sub WS node Client commit A, publish (slow) commit B, publish (fast) B then A B, A — reversed with seq: A=41, B=42 → reorder Each hop was individually ordered; the merge point reordered them
Ordering is lost where independent sources merge.

Resolution #

Assign the sequence number at the one place where a stream’s order is decided — the database transaction, a Redis INCR on the stream’s counter, or the id a log assigns on append — and never renumber downstream. Carry the number in the message envelope through every hop. On the client, keep the last applied number per stream, apply only the next one, buffer briefly for small reorderings, drop duplicates, and resync when a gap does not fill.

// ---------------- Server: assign once, at the ordering point ----------------
import { createClient } from 'redis';
const redis = createClient(); await redis.connect();

// Atomic per-stream counter. INCR is the order: whoever increments first is earlier.
export async function publishOrdered(stream: string, type: string, data: unknown) {
  const seq = await redis.incr(`seq:${stream}`);
  const msg = JSON.stringify({ stream, seq, type, data });
  // Keep a short replay buffer keyed by seq for resume and gap repair.
  await redis.multi()
    .zAdd(`replay:${stream}`, { score: seq, value: msg })
    .zRemRangeByRank(`replay:${stream}`, 0, -1001)      // keep the last 1,000
    .publish(`stream:${stream}`, msg)
    .exec();
  return seq;
}

// ---------------- Client: apply in order, detect gaps ----------------
const REORDER_WAIT_MS = 300;         // how long to wait for a missing seq before repairing

export class OrderedStream<T> {
  private last = 0;
  private pending = new Map<number, T>();
  private gapTimer: ReturnType<typeof setTimeout> | null = null;

  constructor(private apply: (m: T) => void, private repair: (fromSeq: number) => void) {}

  start(fromSeq: number) { this.last = fromSeq; }      // from the snapshot or resume point

  receive(seq: number, msg: T) {
    if (seq <= this.last || this.pending.has(seq)) return;   // duplicate: drop
    this.pending.set(seq, msg);
    // Apply every consecutive message we now hold.
    while (this.pending.has(this.last + 1)) {
      const next = this.pending.get(this.last + 1)!;
      this.pending.delete(this.last + 1);
      this.last += 1;
      this.apply(next);
    }
    if (this.pending.size === 0) { this.clearGap(); return; }
    // Something is missing: give it a moment to arrive, then ask for it.
    if (!this.gapTimer) {
      this.gapTimer = setTimeout(() => {
        this.gapTimer = null;
        if (this.pending.size > 0) this.repair(this.last + 1);   // request replay from the hole
      }, REORDER_WAIT_MS);
    }
  }

  private clearGap() { if (this.gapTimer) { clearTimeout(this.gapTimer); this.gapTimer = null; } }
}

repair asks the server for everything from the missing sequence onward — served from the replay buffer — and if the buffer no longer reaches back that far, the server returns a snapshot instead, and the client restarts the stream from the snapshot’s sequence, as described in reconciling snapshots and deltas over WebSockets.

Where a stream already lives in a log, use the log’s own position instead of a separate counter: a Redis Stream id, a Kafka partition offset, or a database sequence committed in the same transaction as the change. The rule is the same — one authority per stream. The one thing never to do is to let each WebSocket node number the messages it forwards; nodes see different subsets and orders, so their numbers disagree.

Where to assign the sequence number Sequence numbers can be assigned by a Redis counter, a database sequence in the write transaction or a log's own ids, all of which provide a single authority; assigning per WebSocket node or using client timestamps does not. Where to assign the sequence number Single authority Survives restarts Cost Redis INCR per stream yes with persistence 1 round trip DB sequence in the write tx yes yes part of the commit Redis Stream id / Kafka offset yes yes free with the log Per WebSocket node no no wrong answer Client timestamps no n/a no gap detection If the change is committed in a database, number it in the same transaction
One authority per stream, and nobody renumbers downstream.

Edge cases #

Counter vs commit races. Incrementing a Redis counter and then writing to the database leaves a window where sequence 42 is visible before 41’s write commits, or where 41’s write fails and leaves a permanent gap. Either number inside the database transaction, or treat the counter as the commit (write the message to the replay buffer as the durable record).

Permanent gaps. If a sequence number is allocated but its message is never published — the publisher crashed — clients will wait and then repair forever. The server’s repair response must be able to say “42 does not exist, continue from 43”, or send a snapshot.

Stream scope. Per-room streams scale; one global sequence for everything becomes a hot key and couples unrelated data. Choose streams to match the ordering users can observe, as discussed in designing a WebSocket message envelope.

Verification #

Test the client class against scrambled input: shuffle a sequence of 1,000 messages within windows of a few positions, inject duplicates, drop a few, and assert that applied output is strictly increasing, contains no duplicates, and that repair is called exactly for the dropped numbers. On the server, run two publishers concurrently against one stream and confirm the sequence has no duplicates and no holes:

# After a concurrent-publish test: replay buffer must be contiguous.
redis-cli ZRANGE replay:room:42 0 -1 WITHSCORES | awk 'NR%2==0' | \
  awk 'NR>1 && $1!=prev+1 {print "gap after " prev} {prev=$1}'

In production, count client-side repairs per stream per hour. A steady trickle comes from reconnects; a rising rate during stable connections means a publisher or fan-out path is losing messages.

Unrepaired gaps with and without repair requests Outstanding unacknowledged messages over 30 seconds at 100 messages per second with 1 percent ack loss: without redelivery the backlog grows without bound, with a 1 second retry it stays flat. Unrepaired gaps with and without repair requests 100 msg/s, 1% acks lost, 1s ack timeout No retry (leaks) Retry on timeout 0 10 20 30 40 0s 10s 20s 30s
Without repair, every lost message stays lost; with gap detection and replay, the backlog of missing messages drains within a timeout.

Operational checklist #

FAQ #

Doesn’t TCP already keep WebSocket messages in order? #

Only within one connection. Messages that pass through multiple servers, pub/sub layers or reconnects can be reordered, duplicated or dropped before they reach that connection. Sequence numbers restore order end to end.

Can I use timestamps instead of sequence numbers? #

Timestamps can order messages roughly, but clocks differ across servers, ties happen, and a timestamp can never tell you a message is missing. Use a counter for ordering and gap detection; keep timestamps for display.

What if two servers publish to the same room? #

That is exactly why the number must come from a shared authority — a Redis counter, a database sequence or a log — rather than from either server. Both publishers obtain numbers from the same place, so the order is well defined.

How big should the replay buffer be? #

Big enough to cover typical reconnect gaps: a few minutes of a stream’s traffic, or a fixed count such as the last thousand messages. Beyond that, a snapshot is cheaper than a long replay.

Back to Message Delivery Guarantees.