Reconciling snapshots and deltas over WebSockets #

Every live view has two data sources: a snapshot of the current state (from an HTTP request, a server-rendered payload, or a message on subscribe) and a stream of deltas over the WebSocket describing changes since. Stitch them naively and you get one of two bugs. Subscribe after loading the snapshot, and changes made in between are lost — a row that should exist never appears. Subscribe before loading it, and deltas already reflected in the snapshot are applied again — a counter jumps by two, a list item appears twice. The fix is not careful timing; it is a sequence number that places the snapshot and every delta on the same timeline, so the client can tell which deltas are already included, which are new, and when something is missing.

Root cause #

A snapshot and a stream are produced by different mechanisms at different moments. The snapshot reflects the state as of some point in the server’s history; the stream delivers changes as they happen from the moment of subscription. Unless both carry a shared position marker, the client has no way to know how they overlap. Timing-based fixes (“subscribe first, then fetch, and hope the fetch is fast”) shrink the window but never close it, because network latency varies per request and servers process them concurrently.

Deltas make the overlap expensive to get wrong. Applying an absolute value twice is harmless (title = "x" twice is still “x”), but many deltas are relative — increment, append, insert at index — and applying one twice corrupts state in a way no later message corrects.

The overlap problem on one timeline The stream starts before the snapshot is taken, so deltas 41 and 42 are already reflected in the snapshot while delta 43 is new; the client must discard the first two and apply the third. The overlap problem on one timeline subscribe: stream starts delta 41 snapshot taken at seq 42 delta 42 (in snapshot) delta 43 (new) snapshot arrives Positions, not arrival times, decide which deltas the snapshot already contains
Everything at or below the snapshot's sequence is already in it.

Resolution #

Give every delta a monotonic sequence number within its stream, and have the snapshot report the sequence it reflects. The client then follows a fixed procedure: subscribe first and buffer deltas, load the snapshot, discard buffered deltas at or below the snapshot’s sequence, apply the rest in order, and from then on apply each delta only if it is exactly the next number. A delta that skips ahead means one was lost, and the client resyncs by loading a fresh snapshot.

type Delta<S> = { seq: number; apply: (state: S) => S };
type Phase = 'buffering' | 'live' | 'resyncing';

export class Reconciler<S> {
  private phase: Phase = 'buffering';
  private buffer: Delta<S>[] = [];
  private state: S | null = null;
  private seq = 0;

  constructor(
    private loadSnapshot: () => Promise<{ state: S; seq: number }>,
    private onChange: (state: S) => void,
    private maxBuffer = 5_000,        // beyond this, drop the buffer and resync
  ) {}

  // Call after subscribing, so no delta can fall between subscribe and snapshot.
  async start() {
    this.phase = 'buffering';
    this.buffer = [];
    const snap = await this.loadSnapshot();
    this.state = snap.state;
    this.seq = snap.seq;
    // Replay buffered deltas that happened after the snapshot, in order.
    for (const d of this.buffer.sort((a, b) => a.seq - b.seq)) this.applyInOrder(d);
    this.buffer = [];
    if (this.phase === 'buffering') this.phase = 'live';
    this.onChange(this.state);
  }

  receive(d: Delta<S>) {
    if (this.phase !== 'live') {
      this.buffer.push(d);                                  // hold until the snapshot lands
      if (this.buffer.length > this.maxBuffer) void this.resync();
      return;
    }
    this.applyInOrder(d);
    if (this.phase === 'live') this.onChange(this.state!);
  }

  private applyInOrder(d: Delta<S>) {
    if (d.seq <= this.seq) return;                         // already in snapshot or applied
    if (d.seq !== this.seq + 1) {                          // gap: something was lost
      void this.resync();
      return;
    }
    this.state = d.apply(this.state!);
    this.seq = d.seq;
  }

  private async resync() {
    if (this.phase === 'resyncing') return;
    this.phase = 'resyncing';
    this.buffer = [];
    await this.start();                                    // fresh snapshot, keep buffering meanwhile
  }
}

The order in start() is what closes the window: because the subscription already exists when the snapshot request is sent, every change after the snapshot’s sequence is guaranteed to arrive as a delta, either buffered or live. Deltas at or below the snapshot’s sequence are silently dropped, which makes duplicates harmless regardless of whether they are relative or absolute.

The same procedure works whether the snapshot comes from HTTP, from a server-rendered page payload — see using WebSockets in the Next.js App Router — or from the server itself on subscribe. Many protocols simplify it further by letting the client subscribe with its last known sequence (afterSeq), so the server either replays the missing deltas or sends a snapshot when the gap is too large for its buffer, as described in resuming WebSocket sessions after reconnect.

Reconciler phases The reconciler starts buffering after subscribing, becomes live once the snapshot is loaded and buffered deltas are replayed, moves to resyncing when it detects a gap or buffer overflow, and returns to live after a fresh snapshot. Reconciler phases Buffering subscribed, awaiting snapshot Live apply seq+1 only Resyncing gap found, reload snapshot + replay gap or overflow new snapshot A gap is never patched over; it always costs one snapshot
Three phases, and one rule in the live phase: exactly the next sequence, or resync.

Edge cases #

Sequence scope. Sequences must be per stream and assigned at a single point, such as the publisher or a Redis Stream id. If different server nodes number the same stream independently, the client sees gaps that are not gaps and resyncs constantly. See ordering WebSocket messages with sequence numbers.

Snapshot consistency. The snapshot and its sequence must be read atomically: reading the data, then the sequence counter, lets a write slip in between and produces a snapshot that claims to be older than it is. Read both in one transaction, or store the sequence alongside the data.

Resync storms. If a server bug causes gaps for every client at once, every client resyncs at once. Add jitter to resync (a few hundred milliseconds) and a limit on resyncs per minute, falling back to a “data may be stale — reload” banner.

Verification #

Test the race directly with a controllable server. Subscribe, publish deltas 41–43, then return a snapshot at sequence 42, and assert the final state equals snapshot plus delta 43 only. Then drop delta 44 and send 45, and assert a resync occurs:

it('applies only deltas newer than the snapshot', async () => {
  const counter = { state: 0 };
  const r = new Reconciler<number>(
    async () => ({ state: 100, seq: 42 }),          // snapshot already includes 41 and 42
    (s) => { counter.state = s; },
  );
  const started = r.start();
  [41, 42, 43].forEach((seq) => r.receive({ seq, apply: (n) => n + 1 }));
  await started;
  expect(counter.state).toBe(101);                  // only delta 43 applied
});

In production, count resyncs per client session. A healthy system shows resyncs mainly after reconnects; steady resyncs during stable connections mean the server is producing gaps.

Subscribe first, then snapshot The client subscribes before requesting the snapshot, buffers delta 42 that arrives first, receives the snapshot at sequence 42, drops the buffered delta and applies delta 43 when it arrives. Subscribe first, then snapshot Client WebSocket HTTP API subscribe board:7 GET snapshot delta 42 → buffer snapshot at seq 42 drop 42, state current delta 43 → apply Subscribing after the snapshot would have lost any delta published in between
Subscribe, then fetch — never the other way round.

Operational checklist #

FAQ #

Why not just refetch the snapshot on every delta? #

That turns every change into an HTTP request, which does not scale with update frequency, and each refetch can itself race with the next delta. Deltas are cheap; snapshots are for startup and recovery.

Can I use timestamps instead of sequence numbers? #

Timestamps cannot detect gaps — there is no “next” timestamp — and clocks across servers disagree. Use a counter per stream. Timestamps are fine for display and for last-write-wins conflict rules.

What if deltas are idempotent absolute values? #

Then double-applying is harmless and gaps self-heal once the entity changes again, so you can skip gap detection for those fields. Most real streams mix absolute and relative changes, so keep the sequence check.

How large should the server’s replay buffer be? #

Large enough to cover typical reconnect gaps — often a few minutes of traffic per stream. Beyond that, sending a snapshot is cheaper than replaying thousands of deltas.

Back to WebSocket State Sync and Optimistic Updates.