Idempotent WebSocket Message Processing #

The user pressed Send once. The message appears twice — once from the optimistic local write, once because the client resent it after a reconnect and the server had already processed the first copy. Or the accounting variant, which is worse: a “credit 50 tokens” message replayed after a node restart credits 100.

Any real-time system that retries — and every system that survives disconnection retries — delivers some messages more than once. Deduplication is not a nicety bolted on afterwards; it is the property that makes retry safe in the first place.

Root cause #

At-least-once delivery is the only guarantee a distributed system can offer cheaply. Exactly-once, in the strict sense, requires distributed consensus on every message and costs far more than it is worth for real-time traffic. So the practical design is at-least-once delivery plus idempotent processing, which together behave as exactly-once from the outside.

Duplicates enter through four doors, and all four are normal operation rather than bugs:

Client retry after reconnect. The client sent a message, the socket dropped before the ack arrived, and it cannot know whether the server processed it. Resending is the correct behaviour — the alternative is losing the user’s action.

Server replay on resume. A resumed session replays buffered messages, and the client may already have applied some of them, as covered in resuming WebSocket sessions after reconnect.

Redelivery from the bus. A consumer that read from a Redis Stream and crashed before acknowledging will reclaim and reprocess those exact entries on restart.

Cross-region replication. A message forwarded between regions can arrive twice if a link recovers mid-transfer.

The unifying fix is that every message carries an identity assigned by its originator, and every side effect is guarded by that identity.

The same message, delivered twice, applied once A message is processed and acknowledged, the acknowledgement is lost during a disconnect, the client resends the same id, and the server recognises it, skips the side effect and replays the original acknowledgement. The same message, delivered twice, applied once Client Server Dedup store post id=7c2f, 'hello' SET id NX — first time apply side effect ack — lost in the drop reconnect, resend id=7c2f SET id NX — already present replay the stored ack, no side effect Returning the STORED result matters as much as skipping the effect — the client needs an answer, not silence
The duplicate is not rejected. It is answered with the result the first attempt produced.

Resolution #

Deduplicate with an atomic set-if-absent against a bounded window, and store the original result so a duplicate can be answered rather than ignored.

import Redis from 'ioredis';

const DEDUP_WINDOW_SEC = 3600;        // how long a message id is remembered
const redis = new Redis(process.env.REDIS_URL!);

interface Incoming { id: string; t: string; d: unknown; }
type Result = { ok: true; seq: number } | { ok: false; reason: string };

const dedupKey = (userId: string, id: string) => `dedup:${userId}:${id}`;

export async function processOnce(
  userId: string,
  msg: Incoming,
  handler: (m: Incoming) => Promise<Result>,
): Promise<Result> {
  const key = dedupKey(userId, msg.id);

  // Atomic claim. NX means "only if absent", so exactly one concurrent attempt
  // wins even if the duplicate arrives on another node at the same moment.
  const claimed = await redis.set(key, 'pending', 'EX', DEDUP_WINDOW_SEC, 'NX');

  if (!claimed) {
    // Someone already claimed this id. Either it finished — in which case we
    // return its stored result — or it is still running, and we wait briefly.
    const stored = await redis.get(key);
    if (stored && stored !== 'pending') return JSON.parse(stored) as Result;
    await new Promise((r) => setTimeout(r, 50));
    const retry = await redis.get(key);
    if (retry && retry !== 'pending') return JSON.parse(retry) as Result;
    // Still pending: tell the client to retry rather than double-applying.
    return { ok: false, reason: 'in_progress' };
  }

  try {
    const result = await handler(msg);
    // Overwrite the claim with the actual result, keeping the same expiry, so a
    // later duplicate is answered identically instead of being re-processed.
    await redis.set(key, JSON.stringify(result), 'EX', DEDUP_WINDOW_SEC);
    return result;
  } catch (err) {
    // A failed attempt must NOT be remembered as done, or a legitimate retry
    // would be swallowed and the message lost for good.
    await redis.del(key);
    throw err;
  }
}

The del in the catch block is the line people omit and then spend a day debugging. If a handler throws — a transient database error, a timeout — and the claim survives, the client’s retry is deduplicated against a message that never actually succeeded, and the action is silently lost. Failure must release the claim.

Where possible, prefer naturally idempotent operations over a dedup store. Setting a value is idempotent; incrementing is not. Inserting with a unique key is idempotent (the second insert conflicts); appending is not. Reshaping an operation so the database enforces uniqueness is more robust than any application-level window, because it cannot expire:

-- The dedup window can lapse; a unique constraint cannot.
INSERT INTO messages (client_msg_id, room_id, user_id, body)
VALUES ($1, $2, $3, $4)
ON CONFLICT (client_msg_id) DO NOTHING
RETURNING seq;
Which operations need a dedup store A table classifying common operations by whether repeating them is harmless and, when it is not, what guard is required to make retry safe. Which operations need a dedup store Naturally idempotent Guard needed Set a value yes none Insert with unique id yes DB constraint Add to a set yes none Append to a list no dedup by id Increment a counter no dedup by id Send an email no dedup + outbox Charge a card no provider idempotency key Reshaping an operation into the top group is always better than guarding one in the bottom group
Half of real-time traffic is naturally idempotent. The other half is where every duplicate bug lives.

Verification #

Duplicates are rare in development and routine in production, so force them.

# Send the same id twice; the second must be answered, not applied
npx wscat -c ws://localhost:8080/ws \
  -x '{"id":"7c2f","t":"room.post","d":{"body":"hello"}}' \
  -x '{"id":"7c2f","t":"room.post","d":{"body":"hello"}}'

# Concurrent duplicates on two connections — the NX claim must let exactly one win
seq 1 2 | xargs -P2 -I{} npx wscat -c ws://localhost:8080/ws \
  -x '{"id":"9a10","t":"room.post","d":{"body":"race"}}'

# Confirm the window is bounded rather than growing forever
redis-cli --scan --pattern 'dedup:*' | wc -l

Then assert on the data, not the logs: after both sends, the room must contain exactly one message with that id, and the client must have received two acknowledgements with the same sequence number. Track ws_duplicates_suppressed_total in production — a non-zero, stable count means clients are retrying correctly and the guard is working, while a sudden spike usually means a reconnect storm or a bus redelivery worth investigating on its own.

Unacknowledged messages with and without retry Outstanding unacknowledged messages over 30 seconds at 100 messages per second with 2 percent ack loss: without redelivery the backlog grows without bound, with a 5 second retry it stays flat. Unacknowledged messages with and without retry 100 msg/s, 2% acks lost, 5s ack timeout No retry (leaks) Retry on timeout 0 20 40 60 80 0s 10s 20s 30s
Unacknowledged messages over time at 2% ack loss. Every one of those retried messages arrives at the server a second time — which is exactly the duplicate volume the dedup window has to absorb.

Operational checklist #

FAQ #

How long should the dedup window be? #

Longer than the longest interval over which a client might retry, which is bounded by your reconnect backoff ceiling plus the outbox lifetime — for most apps that is minutes rather than hours, so a one-hour window is comfortable. Beyond that, memory cost grows without reducing risk, and anything that could still be retried after an hour belongs behind a database constraint instead, which never expires.

Is exactly-once delivery possible? #

Not over an unreliable network, in the strict sense — the sender can never distinguish “the message was lost” from “the acknowledgement was lost”, so it must either risk losing it or risk sending it twice. What is achievable, and what everyone means in practice, is at-least-once delivery combined with idempotent processing, which produces exactly-once effects. That is the design this page implements.

Should the id come from the client or the server? #

The client, always. The whole point is that the same id survives a retry, and only the originator knows that two attempts are the same attempt. A server-generated id is assigned per delivery, so a retry gets a fresh one and deduplication becomes impossible. Use a UUID generated at the moment of the user action, stored alongside the message in the outbox.

What about ordering — does deduplication break it? #

No, they are orthogonal concerns. Deduplication decides whether a message is applied; ordering decides in what sequence. You need both, and they use different fields: an id for identity and a sequence or version for order, which is why the envelope in message framing and serialization carries both.

Does this need Redis, or can it be in-process? #

In-process works only if the same node always handles a given user’s retries, which is exactly what a reconnect breaks — the client will usually land somewhere else. Use a shared store, or push the guard down into the database with a unique constraint, which has the advantage of surviving a Redis failure and never expiring.

What about idempotency for outbound messages? #

The same idea in the other direction: give every outbound message a stable id so a client can recognise a replayed one and skip it. Combined with the per-entity version check from handling out-of-order WebSocket messages, that makes server replay safe to apply without the client needing to know whether a message is new or repeated.

Back to Message Delivery Guarantees