WebSocket Message Framing & Serialization #

A team ships a collaborative editor. It works beautifully in the office and falls apart on a train: cursors lag by seconds, then jump. The payloads look small — a cursor update is {"type":"cursor","userId":"u_8827","x":412,"y":88}, 58 bytes of JSON. But at 30 updates a second per user, with 12 users in a document, that is 21 KB/s of JSON that must be parsed by the main thread on every client, and roughly 40% of those bytes are the same seven key names repeated forever. The fix is not a faster network. It is deciding, deliberately, what goes on the wire.

This guide covers the layer most real-time systems inherit by accident: the frame the protocol puts around your bytes, the encoding you put inside it, and the envelope you wrap around your domain payload so that version 2 of your client can talk to version 1 of your server. It is the difference between a protocol that survives three years of feature work and one that gets rewritten every time a field changes.

Prerequisites #

You should already have a working connection: the upgrade completing cleanly as described in protocol handshake mechanics, and a transport choice you have justified against the alternatives in WebSocket vs SSE vs WebRTC — SSE, for instance, is text-only, so half of this guide simply does not apply to it. If your payload sizes are large enough that you are considering compression, read backpressure and flow control first: compression changes what bufferedAmount means, and a queue policy tuned before compression will misfire after it.

The frame the protocol adds #

RFC 6455 does not send your bytes; it sends frames. Each frame carries a two-byte minimum header: the first byte holds the FIN bit and a four-bit opcode (0x1 text, 0x2 binary, 0x8 close, 0x9 ping, 0xA pong), the second holds the mask bit and a seven-bit length. Payloads of 126–65535 bytes spend two extra bytes on an extended length field, and anything larger spends eight. Client-to-server frames add a four-byte masking key and XOR the payload with it — a defence against cache-poisoning proxies, not a security feature, and the reason client sends cost slightly more CPU than server sends.

The practical consequence: a server-to-client frame carrying 58 bytes of JSON costs 60 bytes on the wire, while the same update as a 12-byte binary payload costs 14. At 30 updates a second, per user, that difference is real money and real battery.

Core implementation #

The envelope is the part teams get wrong. A raw domain object on the wire — {"x":412,"y":88} — has nowhere to put a message type, a sequence number, or a schema version, so the first time you need one you break every deployed client. Wrap everything in a small, stable envelope from day one, and keep the domain payload opaque inside it.

import { encode, decode } from '@msgpack/msgpack';
import { WebSocket } from 'ws';

const PROTOCOL_VERSION = 2;
const BINARY_THRESHOLD_BYTES = 256;  // below this, JSON's parse speed wins

/** The envelope is the contract; `data` is whatever the type says it is. */
export interface Envelope<T = unknown> {
  v: number;      // protocol version — bump only for incompatible envelope changes
  t: string;      // message type, e.g. 'cursor', 'presence.join'
  s: number;      // monotonic per-connection sequence, for gap detection
  ts: number;     // producer timestamp in epoch ms, for latency measurement
  d: T;           // domain payload, opaque to the transport layer
}

let outboundSeq = 0;

export function send<T>(socket: WebSocket, type: string, data: T): void {
  const envelope: Envelope<T> = {
    v: PROTOCOL_VERSION, t: type, s: ++outboundSeq, ts: Date.now(), d: data,
  };

  // Encode once, choose the frame type from the encoded size. MessagePack wins
  // decisively on repeated-key objects; JSON wins on tiny one-off messages
  // because the browser's JSON.parse is native and MessagePack decode is not.
  const packed = encode(envelope);
  if (packed.byteLength >= BINARY_THRESHOLD_BYTES) {
    socket.send(packed, { binary: true });          // opcode 0x2
  } else {
    socket.send(JSON.stringify(envelope));          // opcode 0x1
  }
}

export function receive(raw: Buffer | string, isBinary: boolean): Envelope | null {
  let envelope: Envelope;
  try {
    // `isBinary` comes from the ws 'message' event — never sniff the first byte,
    // because a valid MessagePack map can start with a printable ASCII byte.
    envelope = isBinary
      ? (decode(raw as Buffer) as Envelope)
      : (JSON.parse(raw.toString()) as Envelope);
  } catch {
    return null;                                     // malformed: drop, count, do not throw
  }

  // Forward compatibility: a newer minor version is readable because unknown
  // fields are ignored; a newer MAJOR envelope version is not.
  if (typeof envelope.v !== 'number' || Math.floor(envelope.v) > PROTOCOL_VERSION) {
    return null;
  }
  if (typeof envelope.t !== 'string' || typeof envelope.s !== 'number') return null;
  return envelope;
}

The s sequence field earns its four bytes the first time a client reports “I missed an update”: a gap in the sequence proves message loss, and its absence proves the bug is elsewhere. It is also what makes handling out-of-order WebSocket messages tractable on the client, and what an at-least-once delivery layer deduplicates on.

Choosing an encoding #

The argument between JSON and a binary codec is usually conducted with adjectives. It is worth conducting it with bytes instead, because for a given message shape the numbers are exact and you can compute them before writing any code.

Take the cursor update from the opening: {"t":"cursor","x":412,"y":88}. As JSON that is 29 bytes — every key name spelled out, every number rendered as decimal text. As MessagePack it is 18: a three-entry map header costs one byte, each single-character key costs two (a length-tagged string), the string cursor costs seven, the value 412 costs three as a 16-bit integer, and 88 costs one because MessagePack encodes small positive integers in a single byte. That is a 38% reduction with no schema, no build step, and no change to your object model.

That 38% is the best case for a small, integer-heavy message, and the saving grows with repetition: a batch of fifty cursor positions in one frame amortises the map headers and drops closer to half. It shrinks, though, for string-dominated payloads — a chat message that is 90% UTF-8 text compresses to nearly the same size in both encodings, because MessagePack stores the same bytes with a shorter length prefix and nothing more.

The cost side is where the decision usually turns. JSON.parse is implemented in C++ inside the JavaScript engine and is extremely fast; a MessagePack decoder is JavaScript, and on a mid-range Android device it is commonly two to four times slower per byte. That reverses the win for small messages: you save 11 bytes of bandwidth and spend more main-thread time than you saved. This is exactly why the implementation above thresholds on encoded size rather than switching wholesale — below a few hundred bytes, native parsing wins; above it, the byte savings dominate and the decode cost is amortised over a much larger payload.

Protobuf sits one step further along the same curve. It drops the keys entirely, encoding field numbers instead, so the cursor update falls to roughly 11 bytes, and its generated decoders are typically faster than a generic MessagePack decoder because the shape is known ahead of time. What you pay is process: a schema file, a code-generation step in every client build, and a versioning discipline where field numbers are permanent and reuse is a bug. That is a good trade for a mobile app with a long release cycle and a poor one for a web app that ships daily and wants to inspect frames in DevTools.

The rule of thumb that survives contact with production: start with JSON and short keys, measure the decode time in a real Performance profile on your worst target device, and only then reach for binary — and when you do, keep both codecs alive behind the threshold so a bad decode never takes the connection down.

Configuration reference #

Edge cases & gotchas #

Fragmentation is invisible until it is not. A large payload may arrive as a sequence of continuation frames; ws reassembles them for you, but the reassembly buffer is bounded by maxPayload and a malicious client can open an unfinished fragment and simply stop, pinning memory. Set maxPayload to a real number and close connections that exceed it.

Control frames can arrive in the middle of a fragmented message. A ping can be interleaved between continuation frames, which is exactly why heartbeat handling must never assume message boundaries. If you have written your own framing on top of a raw socket, this is the case that breaks it.

UTF-8 validation is not free and is not optional. Text frames must contain valid UTF-8, and the receiver is required to fail the connection with 1007 if they do not. If you are shipping bytes that are not text — image tiles, packed floats, a Protobuf message — use a binary frame. Base64-ing binary into a text frame costs 33% more bytes plus an encode and decode on both sides, and is the single most common accidental self-inflicted bandwidth wound in real-time apps.

Compression and encryption interact badly with secrets. permessage-deflate compresses across message boundaries by default via context takeover, which is what makes it effective and also what makes CRIME-style attacks possible when attacker-influenced and secret data share a compression context. If both appear in your stream, disable context takeover — at a substantial compression-ratio cost — or do not compress.

Verification #

Read the actual frames rather than trusting your client library. Chrome DevTools’ Network panel has a Messages tab per WebSocket connection showing opcode, direction, length, and payload — the length column is the number to check, because it is the encoded size, not your object’s notional size.

# Frame-level view including opcodes and lengths, without TLS in the way
tcpdump -i lo -A 'tcp port 8080' | head -40

# What compression is actually negotiated (look for Sec-WebSocket-Extensions)
curl -i -N -H "Connection: Upgrade" -H "Upgrade: websocket" \
  -H "Sec-WebSocket-Version: 13" -H "Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==" \
  -H "Sec-WebSocket-Extensions: permessage-deflate" http://localhost:8080/ws

Then measure the thing that actually matters on the client: main-thread time spent decoding. In DevTools’ Performance panel, a JSON-heavy real-time app shows a repeating sawtooth of JSON.parse blocks; if that sawtooth is a meaningful fraction of your frame budget, binary encoding will help more than any network change. If it is not, stay on JSON and spend your effort somewhere else.

Guides in this area #

• Binary WebSocket Frames with MessagePack — switching to binary frames end to end, including binaryType, the decode cost on mobile, and how to keep both encodings alive during a migration.
• permessage-deflate Compression Trade-offs — the memory-per-socket cost of context takeover, the CPU curve by compression level, and when compression is a net loss.

FAQ #

Should I use MessagePack, Protobuf, or plain JSON? #

Start with JSON, because JSON.parse is native, debuggable, and fast enough for most payload sizes. Move to MessagePack when your bottleneck is bytes on the wire and your objects have many repeated keys — it is a drop-in replacement that keeps the same schemaless shape. Move to Protobuf only when you want a schema enforced across languages and are prepared to pay for a code-generation step in your build; its wire format is smaller still, but the operational cost is real and the debugging story is worse.

Does binary framing break my browser client? #

No, but you must set socket.binaryType = 'arraybuffer' before messages arrive. The default is 'blob', and a Blob has to be read asynchronously with arrayBuffer() or a FileReader before you can decode it, which adds a microtask hop and, worse, allows messages to be processed out of order if you await naively.

What changes for Socket.IO versus raw ws? #

Socket.IO defines its own packet format on top of the WebSocket frame — a type prefix, a namespace, an optional ack id — and handles binary attachments by splitting them into separate frames. You get the envelope for free, at the cost of controlling it: adding a sequence number or a version field means nesting your own envelope inside theirs. On raw ws, the envelope above is the whole protocol and you own every byte.

How do I evolve the protocol without breaking old clients? #

Additive changes only, inside d. New fields must be optional and ignored by older decoders; new message types must be safely skippable by clients that do not recognise t. Reserve v for changes to the envelope itself, and when you must make one, run both versions in parallel for at least one full client-refresh cycle — for a browser app that is however long your longest-lived tab lives, which is usually days, not hours.

Is compression worth enabling for chat-sized messages? #

Usually not. A 200-byte JSON chat message compresses to perhaps 150 bytes, saving 50 bytes, while costing a deflate context of roughly 300 KB of memory per socket. At 20,000 sockets that trade is 6 GB of RAM to save 1 MB per 20,000 messages. Compression pays for itself on large, repetitive payloads — document snapshots, batched updates, long text — and almost never on small ones.

How large should a single WebSocket message be? #

Keep individual frames small enough that one slow message cannot stall the ones behind it — in practice, under about 64 KB for interactive traffic. A WebSocket connection is a single ordered stream, so a 5 MB frame blocks every subsequent message until it finishes transmitting, which on a mobile link can be several seconds of frozen UI. When you genuinely need to move something large, chunk it into sequenced application-level pieces you can interleave with normal traffic, or move it out of band to a plain HTTP request and send only the URL over the socket.

Related #

• Protocol Handshake Mechanics — the upgrade that negotiates extensions and subprotocols before any framing happens.
• WebSocket vs SSE vs WebRTC — where binary framing is available at all, and where it is not.
• WebSocket Backpressure & Flow Control — how encoded size feeds the queue-depth policy on every socket.
• Message Delivery Guarantees — the sequence and id fields in the envelope are what deduplication and replay depend on.
• Handling Out-of-Order WebSocket Messages — the client-side use of the sequence number this envelope carries.

Back to Real-Time Protocol Selection & Architecture