Real-Time Protocol Selection & Architecture #

Picking a real-time transport is an architecture decision, not a library choice. Get it wrong and you inherit silent disconnects behind corporate proxies, half-open sockets that survive a load-balancer restart, or a WebRTC stack you cannot debug at 2 a.m. This guide is for full-stack engineers deciding which protocol to run, then standing it up correctly: the infrastructure that has to be in place first, the WebSocket Upgrade handshake annotated line by line, how to select between WebSocket, Server-Sent Events, and WebRTC, how to degrade when the network fights you, and which signals to watch in production. Read it top to bottom before your first deploy; come back to it the next time a transport “works in dev but not in the data center.”

Real-time transport selection and request path A browser connects through TLS termination and an nginx reverse proxy to a WebSocket, SSE, or WebRTC backend, with fallbacks shown. Browser client WebSocket / fetch TLS + nginx Upgrade headers Backend transports WebSocket: full-duplex SSE: server to client WebRTC: peer media Fallback ladder when Upgrade is blocked WebSocket then SSE then HTTP long-poll Detect at connect, retry with backoff

Infrastructure baseline #

Before any transport works reliably, three things must be true at the edge: TLS terminates correctly, the reverse proxy forwards the Upgrade handshake instead of swallowing it, and the kernel allows enough open file descriptors for your connection count. Skip any one and you will chase a “works locally” ghost.

Serve real-time traffic over TLS (wss://, not ws://). Plaintext WebSocket connections are routinely mangled by intercepting proxies; TLS also stops the Upgrade from being stripped mid-path. Use modern protocol versions and a tight cipher set, and keep the same certificate chain your HTTPS site uses — the Security & TLS Configuration section covers rotation and origin pinning in depth.

# nginx: terminate TLS and forward the WebSocket Upgrade intact
map $http_upgrade $connection_upgrade {
  default upgrade;   # client asked to upgrade -> pass it through
  ''      close;     # plain HTTP request -> let keep-alive close normally
}

server {
  listen 443 ssl;
  server_name realtime.example.com;

  ssl_protocols       TLSv1.2 TLSv1.3;          # no TLS 1.0/1.1
  ssl_ciphers         ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256;
  ssl_prefer_server_ciphers on;

  location /ws/ {
    proxy_pass         http://backend_ws;
    proxy_http_version 1.1;                      # 1.0 cannot carry Upgrade
    proxy_set_header   Upgrade    $http_upgrade; # required header pair
    proxy_set_header   Connection $connection_upgrade;
    proxy_set_header   Host       $host;
    proxy_set_header   X-Real-IP  $remote_addr;
    proxy_read_timeout 3600s;                    # idle sockets must outlive heartbeats
    proxy_send_timeout 3600s;
    proxy_buffering    off;                       # never buffer a streaming socket
  }
}

On the host, raise the descriptor ceiling so a single node can hold tens of thousands of sockets:

# allow many concurrent sockets per worker process
ulimit -n 1048576                         # session limit
echo 'fs.file-max = 2097152' >> /etc/sysctl.conf
sysctl -p

If you are running nginx specifically for upgrade handling, the Browser Compatibility & Polyfills section drills into the exact proxy quirks that break legacy clients.

Core mechanism: the WebSocket Upgrade handshake #

A WebSocket connection is an HTTP/1.1 request that asks the server to switch protocols. The client sends GET with Upgrade: websocket, Connection: Upgrade, and a random Sec-WebSocket-Key. The server proves it understood by hashing that key with the fixed GUID 258EAFA5-E914-47DA-95CA-C5AB0DC85B11, base64-encoding the SHA-1 digest into Sec-WebSocket-Accept, and replying 101 Switching Protocols. After 101, the TCP byte stream is no longer HTTP — it is framed WebSocket data flowing both directions. Subprotocol negotiation, extension headers, and the framing format are dissected in Protocol Handshake Mechanics.

This is also the right moment to authenticate. The Upgrade request still carries cookies and headers, so validate the caller before you complete the handshake — see WebSocket Authentication & Authorization for token verification and origin enforcement at this boundary.

// Node.js: validate the Upgrade manually, then hand off to the ws server.
import { createServer } from 'node:http';
import { WebSocketServer } from 'ws';
import { authenticateUpgrade } from './auth';

// noServer: we own the upgrade handshake so we can reject early.
const wss = new WebSocketServer({ noServer: true });
const server = createServer();

server.on('upgrade', async (req, socket, head) => {
  // The Upgrade is still an HTTP request: headers + cookies are available here.
  const principal = await authenticateUpgrade(req);     // returns null if invalid
  if (!principal) {
    // Reject BEFORE 101 so no WebSocket session is ever created.
    socket.write('HTTP/1.1 401 Unauthorized\r\n\r\n');
    socket.destroy();
    return;
  }

  // Origin check stops cross-site script-driven connections (CSWSH).
  const origin = req.headers.origin;
  if (origin && !isAllowedOrigin(origin)) {
    socket.write('HTTP/1.1 403 Forbidden\r\n\r\n');
    socket.destroy();
    return;
  }

  // ws computes Sec-WebSocket-Accept and emits the 101 response for us.
  wss.handleUpgrade(req, socket, head, (ws) => {
    (ws as any).principal = principal;   // attach identity for later authz
    wss.emit('connection', ws, req);
  });
});

function isAllowedOrigin(origin: string): boolean {
  const ALLOWED = new Set(['https://app.example.com']);
  return ALLOWED.has(origin);
}

server.listen(8080);

The key insight: everything you can check at the HTTP layer — auth, origin, rate limits — must be checked during upgrade, because once 101 is sent the request semantics are gone.

Scaling & architecture: transport selection and fallbacks #

Pick the transport from the data-flow shape, not from familiarity. Use WebSocket when both sides push messages (chat, presence, multiplayer, live editing). Use SSE when the server streams to a mostly passive client (notifications, dashboards, log tails) — it rides plain HTTP, auto-reconnects, and needs no special framing. Use WebRTC data channels only when you need peer-to-peer or sub-50 ms media-grade latency; the signaling and NAT-traversal cost is real. The trade-off matrix lives in the WebSocket vs SSE vs WebRTC Comparison.

Real-time transport decision tree A decision tree: if both sides push frequently use WebSocket; else if peer-to-peer or media-grade latency use WebRTC; else if server-to-client stream use SSE; else plain HTTP fetch. Both sides push frequently? chat, presence, editing yes WebSocket wss:// no Peer-to-peer / media latency? sub-50ms, P2P yes WebRTC data channel no Server-to-client stream only? notifications, dashboards yes SSE event-stream no Plain HTTP / fetch request / response
Runtime fallback ladder A fallback ladder: try WebSocket first; if blocked or timed out try SSE; if that is blocked drop to HTTP long-poll, which always works. Try WebSocket 101 ok? use it blocked Try SSE stream ok? use it blocked HTTP long-poll always works Detect at connect, retry with backoff and jitter

Corporate proxies, captive portals, and some mobile carriers strip the Upgrade header or kill idle sockets. Production clients therefore negotiate a transport at connect time and fall back down the ladder, retrying with exponential backoff and jitter so a regional outage does not produce a thundering-herd reconnect storm.

// Client: detect the best available transport, fall back gracefully.
type Transport = 'websocket' | 'sse' | 'longpoll';

const CONNECT_TIMEOUT_MS = 5_000;   // if 101/stream does not arrive, fall back

function selectTransport(): Transport {
  // Feature detection is necessary but not sufficient: a proxy can still block
  // a runtime-supported transport, so the connect attempt must also time out.
  if (typeof WebSocket !== 'undefined') return 'websocket';
  if (typeof EventSource !== 'undefined') return 'sse';
  return 'longpoll';
}

async function connectWithFallback(url: string): Promise<Transport> {
  const order: Transport[] = ['websocket', 'sse', 'longpoll'];
  const start = order.indexOf(selectTransport());
  for (const transport of order.slice(start)) {
    if (await tryTransport(transport, url, CONNECT_TIMEOUT_MS)) return transport;
    // else loop: the network blocked this one, drop to the next rung.
  }
  throw new Error('No usable real-time transport');
}

Horizontally, a single node cannot hold every connection, so a broadcast on node A must reach a subscriber pinned to node B. The two patterns are sticky sessions (pin a client to one node and keep state in memory) and a shared message bus (publish to Redis or NATS and let every node fan out to its local sockets). Sticky sessions are simpler but make rolling deploys and rebalancing painful; a Pub/Sub bus decouples routing from connection placement. The full fan-out, presence, and delivery-guarantee treatment lives under Scaling Real-Time Infrastructure.

// Cross-node broadcast via Redis Pub/Sub: publish once, every node fans out locally.
import { createClient } from 'redis';

const pub = createClient({ url: process.env.REDIS_URL });
const sub = pub.duplicate();
await Promise.all([pub.connect(), sub.connect()]);

// Each node subscribes to the tenants it currently holds connections for.
await sub.pSubscribe('rt:tenant:*', (message, channel) => {
  const tenantId = channel.split(':')[2];
  fanOutToLocalSockets(tenantId, JSON.parse(message));  // only this node's sockets
});

export async function broadcast(tenantId: string, payload: unknown): Promise<void> {
  await pub.publish(`rt:tenant:${tenantId}`, JSON.stringify(payload));
}

Observability checklist #

Real-time systems fail quietly — a wedged socket looks identical to an idle one until you measure it. Instrument these from day one:

Failure modes #

Failure Symptom Root cause Mitigation
Upgrade stripped at proxy Client gets 200/426, never 101 Reverse proxy not forwarding Upgrade/Connection headers Add the nginx map + proxy_set_header block; verify with curl -i -H 'Upgrade: websocket'
Half-open / zombie socket Server thinks client is connected; no traffic NAT/firewall dropped the flow silently; no heartbeat Server-side ping with a missed-pong threshold, then terminate()
Idle disconnect at the edge Sockets die at a fixed interval (e.g. 60s) Proxy/LB proxy_read_timeout shorter than heartbeat Set proxy idle timeout above the heartbeat interval (3600s)
Reconnect storm Backend CPU and broker saturate after a blip Synchronized client reconnects without jitter Exponential backoff with jitter and a hard retry cap
Slow-consumer back-pressure Node memory climbs, latency rises bufferedAmount grows faster than the client drains Watch ws_send_buffer_bytes; drop or disconnect over a threshold

What goes on the wire #

Choosing WebSocket over SSE or WebRTC settles the transport. It does not settle the far more consequential question of what you put inside it, and that decision is the one teams inherit by accident and pay for later.

The protocol frames your bytes; it does not encode them. RFC 6455 adds a two-byte header to every message, plus two more bytes for payloads between 126 and 65,535 bytes and eight for anything larger, plus a four-byte masking key on client-to-server frames. That overhead is small and fixed. What is not fixed is the payload: a cursor update expressed as {"type":"cursor","userId":"u_8827","x":412,"y":88} spends nearly half its bytes on key names that never change, repeated thirty times a second, per user.

Three decisions follow, in order of how much they matter.

Design an envelope before you design a message. Everything on the wire should be wrapped in a small, stable structure carrying a protocol version, a message type, a monotonic sequence number and a timestamp, with your domain payload opaque inside it. The sequence number is what makes message loss provable rather than suspected; the version is what lets you change the envelope in a year without breaking every deployed client. Retrofitting either into a raw domain object means breaking every client at once.

Pick text or binary per message, not per system. JSON is native, debuggable and extremely fast to parse — JSON.parse is implemented inside the engine. A binary codec such as MessagePack is typically 30–40% smaller on key-heavy objects and two to four times slower to decode in JavaScript, which reverses the win below a few hundred bytes. Thresholding on encoded size gets both: small messages stay readable text, large ones go binary.

Treat compression as a memory decision. permessage-deflate is negotiated per connection and costs roughly 100–300 KB of zlib state per socket depending on window settings, whether or not any message compresses well. At 20,000 connections that is several gigabytes bought to save bytes on payloads that are frequently too small to compress. Set a threshold so only large messages engage it, or leave it off.

Bytes on the wire per update, by transport Wire bytes per update for payloads of 16, 49, 256, 1024 bytes: a WebSocket frame adds two to four bytes, an SSE event eight, and a polling round trip repeats 420 bytes of headers. Bytes on the wire per update, by transport frame/protocol overhead only — polling assumes 420 B of HTTP headers WebSocket frame SSE event HTTP poll 0 B 500 B 1.0k B 1.5k B 16 B 49 B 256 B 1024 B
Wire bytes per update computed from the framing rules of each transport. The frame overhead is negligible either way — the difference between transports at small payload sizes is almost entirely HTTP headers.

The pattern worth internalising is that the transport choice determines your floor and the encoding determines your cost. A WebSocket at 60 bytes per update and a WebSocket at 14 bytes per update are the same protocol decision and a four-fold difference in bandwidth and battery. The framing rules, envelope design, codec migration path and compression arithmetic are covered in WebSocket message framing and serialization.

Surviving hostile networks #

A protocol that works on your laptop and fails at one customer is not a protocol problem; it is a middle-box problem, and it is the single largest source of “works for me” real-time bugs. Corporate networks, mobile carriers and public Wi-Fi all sit between your user and your server in ways the open internet does not.

Five behaviours account for nearly all of it. TLS inspection terminates the connection with a corporate certificate authority and re-originates it, and the appliance decides whether to relay an upgrade at all — many older ones simply do not. Explicit proxies establish wss:// through an HTTP CONNECT, which is commonly restricted to port 443, so a WebSocket on 8080 fails precisely where the equivalent HTTPS request succeeds. Aggressive idle timeouts on the appliance drop connections after 60 to 120 seconds of silence regardless of what your server thinks. Header stripping removes fields the appliance does not recognise, including Sec-WebSocket-Protocol, so a negotiation the client believed happened did not. And content-scanning buffers hold small frames until a buffer fills, adding seconds of latency to a protocol chosen for milliseconds.

Two design decisions eliminate most of these before they occur. Serve on port 443 and route by path rather than by port, which removes the CONNECT restriction and most heuristic filters in one move. And keep the application heartbeat under 30 seconds so no appliance ever sees an idle connection — application ping frames, not TCP keepalive, because an inspecting proxy re-originates the connection and applies its own timer to it.

For what remains, detect rather than guess. A WebSocket that fails within a second or two while an ordinary HTTPS request to the same host succeeds is conclusive evidence that the transport is being blocked rather than the host being unreachable — and that contrast is the thing to put in front of a customer’s IT team, because it turns an unwinnable “your app is broken” conversation into a specific request for one hostname.

Network symptom to protocol-level cause A table mapping the failure a user reports on a restricted network to the middle-box behaviour causing it and the application-side change that resolves or works around it. Network symptom to protocol-level cause Cause Fix you control Never connects inspection drops upgrade allow-list request Fails off port 443 CONNECT restriction serve on 443 Drops every 60-90s appliance idle timeout heartbeat under 30s Wrong codec chosen subprotocol stripped read socket.protocol Seconds of lag scanning buffer fewer, larger messages Four of five have a fix on your side of the connection — only the first genuinely requires the customer's network team
Design for the restricted network and the open one takes care of itself.

Finally, degrade rather than fail. Where the socket genuinely cannot be established, fall back to polling the same API over HTTPS — worse in every way except the one that matters. Server-Sent Events are often proposed as the intermediate step and frequently are not, because the same buffering appliances break streaming responses. The full diagnosis, detection code and IT-facing checklist are in WebSockets behind corporate proxies and firewalls.

Deciding, and then re-deciding #

Transport choices age. The decision that was correct for a notification feed at launch is often wrong once the product grows a collaborative editor, and the cost of not revisiting it is a system that fights its own transport.

Three signals say a re-decision is due. You are sending client-to-server messages over a one-way transport — an SSE feed paired with a POST endpoint for every user action is a WebSocket built badly, paying two connection setups and losing ordering between the two channels. Your message rate has crossed into the tens per second per client, where per-message HTTP overhead stops being noise and starts being most of your bandwidth. Or your latency requirement has tightened below what a relayed transport can deliver — at which point the question is not WebSocket versus SSE but whether a peer-to-peer data channel is warranted, with all the signalling and NAT-traversal complexity that implies.

Equally, three signals say you over-chose. A WebSocket carrying one update a minute is a persistent connection, a heartbeat, a reconnect state machine and a sticky-session requirement bought to replace a poll. A WebSocket used exclusively server-to-client is an SSE feed that gave up automatic reconnection and browser-native resume for nothing. And a WebSocket in an environment where a meaningful fraction of users sit behind inspecting proxies may be delivering a worse experience than plain polling would, for the subset that matters most.

The honest way to hold this is that transport selection is a constraint satisfaction problem with four axes — direction, frequency, latency tolerance and network hostility — and that only the last one is outside your control. Write down where your product sits on each, revisit it when any of them changes by an order of magnitude, and treat “we already use WebSockets everywhere” as a description of the current state rather than an argument.

Explore this area #

FAQ #

Should I just use Socket.IO instead of choosing a transport myself? #

Socket.IO bundles a transport ladder (WebSocket with HTTP long-poll fallback) and a reconnection layer, which is convenient. The cost is a non-standard wire protocol, a heavier client, and harder debugging when something goes wrong at the network layer. If you need raw control, browser-native WebSocket plus the ws package on the server keeps the protocol transparent and easy to inspect with curl and DevTools. Choose Socket.IO for speed-to-ship; choose raw ws for control and observability.

How do I confirm my reverse proxy is forwarding the Upgrade correctly? #

Send a handshake by hand and look for 101:

curl -i -N -H 'Connection: Upgrade' -H 'Upgrade: websocket' \
  -H 'Sec-WebSocket-Version: 13' \
  -H 'Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==' \
  https://realtime.example.com/ws/

A 101 Switching Protocols with a Sec-WebSocket-Accept header means the proxy passed the upgrade through. A 200, 400, or 426 means a hop is stripping or rejecting the headers — check every proxy in the path.

Does this work behind AWS ALB? #

Yes. ALB supports WebSocket upgrades natively and keeps the connection on a single target, but its default idle timeout (60s) is shorter than most heartbeat intervals, so raise the idle timeout or ping more often. For SSE the same idle-timeout rule applies. WebRTC media does not flow through the ALB at all — only the signaling channel does.

When is WebRTC actually worth the complexity? #

Only when you need peer-to-peer data paths or media-grade latency that a server relay cannot provide — live audio/video, low-latency game state between players, or screen sharing. For everything server-mediated (chat, dashboards, notifications), a WebSocket or SSE connection is simpler to deploy, authenticate, and observe.

Can I run WebSockets over HTTP/2? #

Not with the classic upgrade, which is HTTP/1.1 only. HTTP/2 defines extended CONNECT in RFC 8441 for the same purpose, and browsers will use it where both ends support it — but many intermediaries do not, so a CDN or proxy that negotiates h2 to your origin without extended CONNECT breaks upgrades in ways that look like a routing bug. Forcing HTTP/1.1 in a curl probe is what distinguishes that case.

How should I version a real-time protocol? #

Two levers, used for different things. The subprotocol name negotiated at the handshake (v2.chat against v1.chat) selects the contract before the first byte of application data, so a server can serve both generations at once with no feature detection. Inside that contract, evolve additively: new fields optional, new message types safely ignorable, and removals only after a full client-refresh cycle — which for a browser app is however long your longest-lived tab lives.

What is the smallest useful envelope? #

Four fields: a protocol version, a message type, a monotonic sequence number and a timestamp, with the domain payload opaque underneath. The sequence is what makes loss provable rather than suspected, the timestamp is what makes end-to-end latency measurable, and the version is what lets you change the envelope later. Adding any of them retrospectively means breaking every deployed client at once.

Back to Real-Time WebSocket Engineering