Testing WebSocket reconnection with network chaos #

The reconnect logic has unit tests with a mocked socket, and they all pass. Then production traffic crosses a flaky hotel Wi-Fi and a corporate proxy that resets idle connections, and users report duplicated messages, a spinner that never stops, and a chat that silently stopped updating. Mocks test the code paths you thought of: a clean close event, a scripted error. Real networks produce failures that mocks rarely model — half-open connections that never close, latency spikes that trip timeouts halfway through a resume, a reset in the middle of a replay. Network chaos testing runs the real client against the real server through a proxy that injects those failures on command, and asserts on what users care about: that the client recovers, in bounded time, with no lost or duplicated data.

Root cause #

Reconnection bugs live in timing and state, which is exactly what mocks flatten. A mocked socket closes instantly and cleanly; a real network can blackhole packets so the client sees nothing for minutes, the failure described in detecting half-open WebSocket connections. A mock delivers messages synchronously; a real network delays some and not others, so a resume request can race with late messages from the old connection. A mock never fails during recovery; a real network can drop the new connection while the client is replaying its offline queue, which is where duplicates and losses are born.

The only reliable way to find these bugs is to reproduce the network behaviour itself, deterministically, in a test that can run in CI.

Chaos test harness A chaos test harness runs a real browser through Toxiproxy to a real WebSocket server, with a test script toggling network faults, publishing messages and asserting on recovery time, gaps, duplicates and UI state. Chaos test harness Browser (Playwright) real client code, real WebSocket API client Toxiproxy injects latency, bandwidth limits, resets, timeouts on command chaos WebSocket server real server, real resume and replay logic server Test script toggles toxics, publishes messages, asserts on UI and logs control Assertions recovery time, no gaps, no duplicates, UI state outcome Everything is real except the network, which is deliberately bad
Real client, real server, a programmable bad network in between.

Resolution #

Put Toxiproxy between the browser and the WebSocket server. It is a TCP proxy with an HTTP API for adding toxics — latency, bandwidth limits, connection resets, timeout (blackhole: stop forwarding without closing) — at any moment. Drive the client with Playwright, publish a numbered sequence of messages from the server side during the chaos, and assert that the UI ends up with every message exactly once, within a recovery budget.

// chaos.spec.ts — Playwright test against a real server behind Toxiproxy.
import { test, expect } from '@playwright/test';

const TOXI = 'http://localhost:8474';          // Toxiproxy API
const PROXY = 'ws_upstream';                   // proxy listening on :18080 → server :8080
const RECOVERY_BUDGET_MS = 10_000;

async function toxic(type: string, attributes: object, name = type) {
  await fetch(`${TOXI}/proxies/${PROXY}/toxics`, {
    method: 'POST', headers: { 'content-type': 'application/json' },
    body: JSON.stringify({ name, type, stream: 'downstream', attributes }),
  });
}
const clearToxics = async () => {
  const list = await (await fetch(`${TOXI}/proxies/${PROXY}/toxics`)).json();
  await Promise.all(list.map((t: any) => fetch(`${TOXI}/proxies/${PROXY}/toxics/${t.name}`, { method: 'DELETE' })));
};

declare function publish(room: string, n: number): Promise<void>;   // server-side helper: sends msg #n

test.afterEach(clearToxics);

test('recovers from a blackholed connection without gaps or duplicates', async ({ page }) => {
  await page.goto('/room/chaos?ws=ws://localhost:18080/ws');
  await expect(page.getByTestId('status')).toHaveText('live');

  for (let n = 1; n <= 20; n++) await publish('chaos', n);
  await toxic('timeout', { timeout: 0 });               // blackhole: no data, no close — half-open
  for (let n = 21; n <= 40; n++) await publish('chaos', n);   // published while the client is cut off

  // Client should detect the dead socket (heartbeat) and show it is reconnecting.
  await expect(page.getByTestId('status')).toHaveText('reconnecting', { timeout: 60_000 });
  await clearToxics();                                  // network comes back
  const t0 = Date.now();
  await expect(page.getByTestId('status')).toHaveText('live', { timeout: RECOVERY_BUDGET_MS });
  for (let n = 41; n <= 50; n++) await publish('chaos', n);

  // Every message exactly once, in order.
  await expect(page.getByTestId('message')).toHaveCount(50);
  const seqs = await page.getByTestId('message').evaluateAll((els) => els.map((e) => Number(e.getAttribute('data-seq'))));
  expect(seqs).toEqual(Array.from({ length: 50 }, (_, i) => i + 1));
  expect(Date.now() - t0).toBeLessThan(RECOVERY_BUDGET_MS);
});

test('survives a reset in the middle of resume', async ({ page }) => {
  await page.goto('/room/chaos2?ws=ws://localhost:18080/ws');
  await expect(page.getByTestId('status')).toHaveText('live');
  await toxic('reset_peer', { timeout: 0 });            // kill the connection now
  for (let n = 1; n <= 30; n++) await publish('chaos2', n);
  await clearToxics();
  await toxic('latency', { latency: 800, jitter: 400 }); // slow, jittery recovery path
  await toxic('reset_peer', { timeout: 1500 }, 'reset2'); // …then reset again mid-resume
  await clearToxics();
  await expect(page.getByTestId('message')).toHaveCount(30, { timeout: 30_000 });
});

The first test exercises the path mocks almost never cover: a connection that stays “open” but carries nothing. The client must detect it through its heartbeat and liveness logic, reconnect, and resume from the last sequence so messages 21–40 are replayed — the protocol in resuming WebSocket sessions after reconnect. The second injects a failure during recovery, which is where duplicate and gap bugs hide. For faster, lower-fidelity checks, Playwright’s context.setOffline(true) simulates the browser going offline, and a fake socket with fake timers covers backoff arithmetic, as in mocking WebSockets in Vitest.

Faults worth injecting, and what they catch Useful faults to inject: a blackhole reveals missing liveness checks, a peer reset exercises reconnect and resume, latency with jitter exposes tight timeouts and ordering bugs, bandwidth limits exercise backpressure, and a fault during resume exposes duplicates and gaps. Faults worth injecting, and what they catch Toxic Catches Blackhole (timeout 0) half-open socket missing liveness checks reset_peer abrupt 1006 close reconnect + resume path Latency + jitter slow, reordered delivery timeouts too tight, ordering bugs Bandwidth limit slow consumer backpressure, eviction handling Fault during resume double failure duplicates and gaps The double-failure row finds the bugs users actually report
Each fault targets a different piece of recovery logic.

Edge cases #

Heartbeat timing in tests. A blackhole is only detected after the client’s liveness timeout, which in production may be tens of seconds. Make timeouts configurable and shorten them in the test build, or the suite becomes slow — but keep at least one test with production values to catch interactions with real proxies.

Toxics apply per direction. Toxiproxy toxics target the upstream or downstream stream. Blackholing only downstream simulates a connection where the client can send but never hears back — a realistic and nasty failure mode; test both directions.

Server-side state. Chaos tests must use a server whose replay buffer and sequence numbers behave like production. A test server that resets state between reconnects will pass tests the real one fails.

Verification #

Make the suite prove its own value. Temporarily break the client — remove the sequence-based deduplication, or disable the liveness timeout — and confirm the corresponding chaos test fails. Run the suite repeatedly in CI (at least nightly) since timing bugs are intermittent, and record recovery times per test to catch slow regressions:

# Start the harness locally.
docker run -d --name toxiproxy -p 8474:8474 -p 18080:18080 ghcr.io/shopify/toxiproxy
curl -s -X POST localhost:8474/proxies -d '{"name":"ws_upstream","listen":"0.0.0.0:18080","upstream":"host.docker.internal:8080"}'
npx playwright test chaos.spec.ts --repeat-each=5
The blackhole test, as the client experiences it The blackhole starts at zero seconds; the client's liveness timeout detects it at twenty seconds and shows reconnecting; the network is restored at twenty-eight seconds, the client reconnects at twenty-nine and resumes from sequence twenty, and messages twenty-one to forty are replayed. The blackhole test, as the client experiences it socket looks open, carries nothing blackhole on (0 s) liveness timeout: reconnecting (20 s) network restored (28 s) reconnected, resume afterSeq=20 (29 s) 21–40 replayed, UI complete (29.5 s) Without a liveness timeout, the client would sit in the first band forever
The test fails if any band is missing or too long.

Operational checklist #

FAQ #

How do I test WebSocket reconnection realistically? #

Put a fault-injecting proxy such as Toxiproxy between a real browser and a real server, inject failures such as blackholes, resets and latency during a test, and assert on recovery time and message integrity rather than on internal function calls.

Isn’t Playwright’s offline mode enough? #

It is useful for quick checks, but it simulates the browser knowing it is offline, which triggers clean error paths. Real failures often look like a connection that is still open but silent, which only a proxy that stops forwarding reproduces.

What should chaos tests assert? #

That the client recovers within a time budget, that every message published during the failure is delivered exactly once and in order after recovery, and that the UI reflects the connection state correctly throughout.

How do I keep chaos tests from being flaky? #

Wait on observable state (status indicators, message counts) instead of fixed sleeps, make timeouts configurable, clear toxics after every test, and run the suite repeatedly to separate real intermittent bugs from test timing issues.

Back to Testing Real-Time Frontends.