WebSocket
Master WebSocket: real-time bidirectional communication, protocol details, and implementation patterns.
What is WebSocket?
WebSocket is a protocol for a long-lived, full-duplex, framed connection between a client and server. It begins as an HTTP upgrade request, then both sides exchange text, binary, continuation, ping, pong, and close frames over one underlying connection.
WebSocket matters for chat, collaboration, live markets, games, device control, and other products where either side must send low-latency events without opening a new HTTP request for each message. Persistent connections create new obligations: connection admission, per-peer memory, fan-out, backpressure, liveness, reconnect, resume, authorization, deployment drain, and regional recovery.
The core invariant is every accepted connection and application message belongs to one authenticated identity, schema, subscription version, and delivery state. An open socket does not prove a message was processed, and a reconnect must not silently hide missed or duplicated events.
Decide whether a persistent bidirectional channel is justified
Bidirectional + framed
WebSocket
Use when client and server both send independent events, binary frames matter, or low-latency interaction cannot wait for repeated requests.
Server to browser
Server-Sent Events
Use when the browser sends ordinary HTTP requests but receives one ordered text event stream with built-in event IDs and reconnection support.
Request driven
Polling or long polling
Use when update frequency is low, cache and stateless infrastructure matter, or simple retry semantics outweigh immediate push.
Do not choose WebSocket only to reduce headers. Measure product freshness, message direction, client support, intermediary behavior, stateful capacity, and the recovery contract.
Establish a trusted connection
1 TLS
Secure the HTTP request
Use
wss, validate host and transport policy, and bound header size, handshake time, and connection attempts.2 Identity
Authenticate and check origin
Bind user, tenant, device, session, and browser origin where relevant without placing long-lived secrets in logged query strings.
3 Subprotocol
Negotiate capability
Accept only supported WebSocket version, application subprotocol, compression policy, and message-schema family.
4 Capacity
Admit into a gateway shard
Enforce account, IP, tenant, region, subscription, and total connection quotas before allocating persistent state.
5 101
Return the upgrade
Complete the protocol acceptance and attach one traceable connection ID used across broker, gateway, and client telemetry.
Reauthorize sensitive operations during the connection. Identity, role, membership, and resource policy can change after the handshake.
Size connection state and fan-out independently
Size persistent connections and fan-out separately
A gateway can fit its sockets and still fail on outbound amplification. Change connection, message, room, coalescing, and per-node assumptions to expose the real bottleneck.
Gateway verdict
Connections and amplified sends remain below the modeled limits
Persistent connection state is the controlling resource. Add shards, lower per-peer state, or rebalance connection admission.
12,500
63% of safe limit
10,000/s
20,000 active senders
168,000/s
8,400/s per node
0.46 Gbit/s
1% of per-node network limit
Admit
Authenticate, validate origin and subprotocol, enforce account and IP connection quotas, then assign a gateway shard.
Fan out
Partition channels through a broker, coalesce replaceable events, and bound every per-connection send queue.
Drain
Stop new admission, preserve resume cursors, close with retry guidance, and jitter reconnect across healthy shards.
The model excludes TLS overhead, broker replication, kernel buffers, compression CPU, retransmission, uneven rooms, and reconnect traffic. Measure representative gateway processes.
Budget each gateway resource
- Sockets, file descriptors, TLS state, parser state, subscriptions, send queues, timers, and application session data
- Inbound messages, validation CPU, broker publishes, channel lookups, and authorization checks
- Outbound fan-out, serialization, compression, encryption, network bandwidth, retransmission, and slow-client queues
- Connection churn, handshake CPU, authentication dependencies, broker resubscription, and reconnect admission
Partition channels and broker traffic so one popular topic does not traverse every gateway. Measure skew and keep durable user or room state outside one connection process.
Define a versioned application message envelope
WebSocket frames are not business messages. A logical message can span continuation frames, and one frame can contain any application-defined bytes. Use a bounded schema with message ID, type and version, payload, trace context, and any sequence or acknowledgement fields required by the protocol.
Validate before routing
- Reject unknown types, versions, fields, encodings, oversized payloads, and invalid tenant or resource identifiers.
- Bound decompression ratio, nesting, arrays, strings, message rate, and expensive authorization checks.
- Keep transport ping/pong separate from application acknowledgement and business commit.
- Use idempotency keys and durable deduplication for messages that can trigger external effects.
Apply backpressure per connection and topic
An open TCP connection can keep accepting application work faster than one client can render or one gateway can send. Unbounded queues convert a slow client into memory exhaustion and stale delivery.
Coalesce
Replace queued presence, cursor, price, or position state with the newest version when intermediate values have no lasting meaning.
Drop
Discard explicitly replaceable events with counters and a resync signal; never silently drop durable commands or chat messages.
Disconnect
Close a persistently slow peer with a typed reason and retry guidance before its queue threatens other tenants.
Resync
Send a compact snapshot pointer or require the client to fetch current authority before resuming live events.
Set queue limits by messages, bytes, and age. Feed broker demand from admitted downstream capacity rather than treating the gateway as an infinite sink.
Reconnect without gaps or a retry storm
Loading reconnect lab
Preparing recovery scenarios...
Make resume a protocol feature
- Give every replayable event a stable stream or channel sequence and event ID.
- Persist the last application-acknowledged cursor at the boundary that must survive restart.
- Reconnect with exponential backoff, jitter, cancellation of superseded attempts, and server retry hints.
- Authenticate again, restore authorized subscriptions, and request replay after the cursor.
- Detect cursor expiry; load an authoritative versioned snapshot before joining later live events.
- Deduplicate the overlap between replay and current traffic, then mark the client current.
Scale gateways, broker, and state as separate planes
One durable event reaches connected clients
The durable stream owns replay; gateways own current connection delivery and bounded queues.
Commit
Authoritative service
Commit the business result and a stable event or outbox record once.
Retain
Partitioned broker
Order and retain events by channel or entity so replay and lag remain measurable.
Fan out
Gateway shard
Consume assigned partitions, map authorized subscribers, and enforce per-peer queues.
Acknowledge
Client
Apply the event, advance its cursor durably, and expose whether the local view is current.
Sticky routing can reduce movement but cannot become the only copy of durable session, subscription, or cursor state. Rebalancing, gateway loss, and regional failover must reconstruct delivery from independent evidence.
Deploy and drain without synchronized reconnect
Rolling gateway procedure
- Start replacements, warm broker subscriptions and caches, and prove handshake capacity before draining old nodes.
- Stop new admission to one small cohort while existing connections continue.
- Send a typed close reason and randomized retry window; retain cursor and subscription state.
- Watch reconnect rate, handshake latency, authentication load, broker lag, queue depth, and error rate before continuing.
- Force-close only after the drain deadline, and retain a rollback path that does not disconnect the whole fleet again.
Regional recovery additionally requires global routing, independent gateway and broker capacity, replicated or rebuildable subscription state, event-retention review, and cursor semantics across the authority change.
Secure the long-lived channel
Handshake abuse
Rate-limit attempts, bound authentication work, validate origin and subprotocol, and avoid resource allocation before admission succeeds.
Message abuse
Validate schema and authorization per action, cap message size and rate, and isolate expensive handlers from the gateway event loop.
Cross-tenant fan-out
Partition subscription indexes by tenant and recheck room or resource membership before publishing sensitive data.
Compression risk
Bound decompression, CPU, and memory; disable or scope compression for secrets or untrusted high-ratio payloads where appropriate.
Operate from connection and delivery evidence
Monitor gateways
- Connections by region, zone, gateway, tenant, subprotocol, version, age, churn, and close code
- Handshake attempts, authentication latency, rejection reason, origin, rate limit, and admission saturation
- Inbound/outbound messages and bytes, parse failure, authorization denial, handler latency, and per-type rate
- Send queue messages, bytes, age, coalesced and dropped events, disconnects, and slow-client distribution
- Broker partition lag, hot topics, fan-out amplification, replay rate, cursor gaps, duplicate events, and resyncs
Test the recovery path
- Slow readers, paused clients, half-open connections, missed pong, oversized and fragmented frames, and invalid compression
- Gateway crash, broker pause, partition rebalance, authentication dependency failure, and session-store loss
- Rolling deploy, mass network flap, reconnect storm, cursor expiry, snapshot mismatch, regional failover, and failback
A production WebSocket system does not promise that a socket stays open forever. It promises bounded connection state, honest message outcomes, controlled fan-out, and a tested path back to a coherent client view.