Load testing WebSockets with Artillery #

Your team already uses Artillery for HTTP load tests, defined in YAML and run in CI, and now the real-time service needs the same treatment. Artillery supports WebSockets through its ws engine (and Socket.IO through a dedicated engine), which means the real-time scenarios can live next to the HTTP ones, share configuration and reporting, and run in the same pipelines. Its model differs from script-first tools: instead of writing a loop per virtual user, you describe arrival phases — how many new users start per second — and flows — what each user does. Getting realistic results depends on translating a real-time workload into that model correctly, especially the long idle holds that make WebSocket load different from HTTP load.

Root cause #

Artillery’s arrival model comes from HTTP testing: users arrive, run a flow, and leave. A flow that connects, sends one message and ends measures handshakes, not the sustained concurrency that dominates real WebSocket costs. The number of concurrent connections at any moment is roughly arrival rate × flow duration, so to hold 20,000 connections with a 10-minute flow you need about 33 arrivals per second for ten minutes — and a flow that actually stays connected for those ten minutes, mostly idle, the way real clients do.

The second trap is measurement. Artillery reports handshake and message-send metrics out of the box, but not the latency of messages received from others, which is what users experience in chat, dashboards and collaboration. That needs a small custom processor that timestamps outgoing messages and records the delay on incoming ones.

Concurrency from arrival rate and flow duration At fifty arrivals per second, a one-minute flow holds three thousand concurrent connections, a ten-minute flow thirty thousand and a thirty-minute flow ninety thousand. Concurrency from arrival rate and flow duration steady state after one flow duration arrivals per second concurrent connections 0 25k 50k 75k 100k 1 min flow 30k 10 min flow 90k 30 min flow
Concurrency is arrival rate times flow duration — short flows never build realistic load.

Resolution #

Configure phases that ramp arrivals up and hold them, a flow that connects, joins, then loops think-send cycles for a long time, and a processor that records delivery latency as a custom metric. Artillery’s ws engine keeps the connection open for the whole flow.

# ws-load.yml — run: npx artillery run ws-load.yml
config:
  target: "wss://staging.example.com"
  processor: "./latency.js"
  phases:
    - name: ramp
      duration: 600          # 10 min
      arrivalRate: 5
      rampTo: 40             # up to 40 new users per second
    - name: hold
      duration: 900          # 15 min at a steady arrival rate
      arrivalRate: 40
  ws:
    subprotocols: ["chat.v2"]
  ensure:                    # fail the run (non-zero exit) when these are breached
    thresholds:
      - "vusers.failed": 100
      - "ws.delivery_latency.p95": 250
      - "ws.delivery_latency.p99": 800

scenarios:
  - name: chat-member
    engine: ws
    flow:
      - connect: "/ws?room={{ $randomNumber(1, 200) }}&ticket=loadtest"
      - function: "attachLatencyRecorder"      # starts listening for incoming messages
      - send: '{"type":"room.join"}'
      - loop:
          - think: 30                          # mostly idle, like a real client
          - function: "stampedMessage"         # sets vars.msg with sentAt
          - send: "{{ msg }}"
        count: 20                              # ~10 minutes connected per user
// latency.js — custom processor for Artillery's ws engine
module.exports = {
  attachLatencyRecorder(context, events, done) {
    // The engine exposes the socket on the context once connected.
    context.ws.on('message', (raw) => {
      try {
        const msg = JSON.parse(raw);
        if (msg.type === 'chat.message' && msg.sentAt && msg.from !== context.vars.$uuid) {
          events.emit('histogram', 'ws.delivery_latency', Date.now() - msg.sentAt);
          events.emit('counter', 'ws.messages_received', 1);
        }
      } catch { /* non-JSON frames are ignored */ }
    });
    context.ws.on('close', (code) => {
      if (code !== 1000) events.emit('counter', 'ws.unexpected_close', 1);
    });
    return done();
  },
  stampedMessage(context, events, done) {
    context.vars.msg = JSON.stringify({ type: 'chat.send', sentAt: Date.now(), from: context.vars.$uuid, text: 'load' });
    return done();
  },
};

The double-brace expressions in the YAML are Artillery’s own template syntax, evaluated per virtual user; they are not part of your application’s protocol. The ensure thresholds make the run fail in CI when delivery latency or failures regress, turning the test into a gate rather than a report. For Socket.IO backends, use engine: socketio with emit steps instead; it speaks Socket.IO’s protocol, including acknowledgements, so the same flow structure applies.

For large tests, one machine is not enough: each virtual user holds a connection, and a single Node process running Artillery tops out well before a production fleet does. Artillery can distribute a test across many workers on cloud infrastructure (for example AWS Fargate or Lambda), which also spreads the source IPs — useful when your edge applies per-IP limits, as discussed in rate limiting WebSocket handshakes.

One Artillery virtual user An Artillery virtual user connects, joins a room, thinks for thirty seconds, sends a timestamped message that the server broadcasts to other virtual users, which record the delivery latency, and repeats twenty times before closing. One Artillery virtual user Artillery VU Server Other VUs connect /ws?room=42 room.join think 30 s chat.send (sentAt) broadcast to room histogram: now − sentAt loop × 20, then close Latency is recorded by the receivers, not the sender
Connect, idle, send, measure — for as long as a real user would stay.

Edge cases #

Template collisions. Artillery templates use double braces; if your messages legitimately contain them, build the payload in a processor function rather than inline in YAML.

Clock skew in distributed runs. Latency measured as now − sentAt across different worker machines includes clock differences. Either keep sender and receiver on the same worker (rooms scoped per worker) or have the server stamp messages at publish time.

Engine versions. The ws engine’s API for accessing the underlying socket in processors has changed across Artillery versions. Pin the version in your project and check the processor hook against its documentation when upgrading.

Verification #

Check that the test produces the load you intended before reading its results. During the hold phase, the server’s connection gauge should sit near arrival rate × flow duration; if it is far lower, flows are ending early (errors, timeouts, or a missing loop). Then confirm the thresholds can fail by injecting latency into the server’s broadcast path:

npx artillery run --output report.json ws-load.yml
npx artillery report report.json          # HTML report with custom histograms
# In parallel, on the server: connections should plateau near 40/s × 600 s ≈ 24k.
curl -s http://staging-node:9464/metrics | grep ^ws_connections_open

Compare runs over time: a rising p95 delivery latency at the same concurrency is a performance regression worth blocking a release for.

Artillery vs k6 for WebSocket tests Artillery defines tests in YAML with JavaScript processors, uses an arrival-rate model and has a dedicated Socket.IO engine; k6 uses JavaScript scripts with either virtual-user or arrival-rate models; both support custom latency metrics and distributed runs. Artillery vs k6 for WebSocket tests Artillery k6 Test definition YAML + JS processors JavaScript scripts Load model arrival rate VUs or arrival rate Socket.IO support dedicated engine manual protocol Custom latency metrics processor histograms Trend metrics Distributed runs cloud workers cloud / operator Choose the tool your team already runs; the modelling rules are the same
Two good tools; the realism comes from how you model users.

Operational checklist #

FAQ #

Does Artillery support WebSockets? #

Yes, through its ws engine for raw WebSockets and a socketio engine for Socket.IO servers. Flows can connect, send, wait and run custom JavaScript against incoming messages.

Why does my Artillery test show far fewer connections than expected? #

Concurrency equals arrival rate times flow duration. Short flows — connect, send, disconnect — never accumulate connections. Add think time and loops so each virtual user stays connected as long as a real user would.

How do I measure message latency with Artillery? #

Put a timestamp in outgoing messages from a processor function, and in a listener on incoming messages emit a histogram of the elapsed time. The histogram then appears in the report and can be used in thresholds.

Should I use Artillery or k6? #

Both handle WebSocket load well. Artillery suits teams that like declarative YAML and need Socket.IO support; k6 suits teams that prefer scripting everything in JavaScript. The workload modelling principles are identical — see load testing WebSockets with k6.

Back to Load Testing & Capacity Planning.