Day 64: The Offer/Answer Dance: Exchanging JSON-RPC SDP payloads over WebSockets

A junior engineer discovers spring-webrtc or reaches for a JSON library to relay SDP blobs. The code looks like this:

At 10 users in a demo this is fine. At 10,000 concurrent voice calls, you have a problem you cannot see until production explodes.

Spring’s SimpMessageHeaderAccessor wraps every message in a Map<String, Object> of header copies. Each SDP blob is 2–4 KB of ASCII. Each JSON serialization produces intermediate String and byte[] objects. GC pressure begins to spike before you hit 500 concurrent sessions. More critically: the session correlation model (sessionId → targetId) lives in a HashMap behind a @Service singleton. The first time two threads race on a state transition — offer arriving while an answer is already in flight — you get a corrupted session with no error, just a silent WebRTC failure on the client.

SDP state is a two-party handshake with strict ordering:

Caller → OFFER → Server → forward to Callee
Callee → ANSWER → Server → forward to Caller
Both   → ICE CANDIDATES (trickle) → Server → cross-forward

Three things break at scale with naive implementations:

1. String Interning and Heap Retention

SDP contains repeated attribute prefixes (a=, m=, c=). If you parse into Map<String, String> and retain these maps per session, the String pool bloats. With 1 million sessions at 3 KB each, retained heap from SDP strings alone exceeds 3 GB. The JVM’s G1 collector begins spending 40%+ of wall time reclaiming this.

2. Unsynchronized State Transitions

The session lifecycle is: IDLE → OFFERING → ANSWERING → ESTABLISHED. If a ConcurrentHashMap.put() isn’t paired with a CAS on the state field, two simultaneous answers for the same session both “succeed” and one is silently dropped.

3. Trickle ICE Fan-Out

After SDP exchange, each peer sends 3–10 ICE candidates. These are small messages but they arrive in a burst. A thread-per-connection model spawns hundreds of threads just for ICE candidate forwarding. Under Virtual Threads this is invisible, but if you’re on platform threads (the Spring default until explicitly configured otherwise), you hit OS thread limits fast.

The Flux SDP Exchange system has three components.

SdpSession — Sealed State Machine

A sealed interface with record variants for each lifecycle state. State transitions use VarHandle CAS to prevent races without synchronized blocks.

No inheritance, no nullable fields, no instanceof chains. Pattern matching in switch expressions handles dispatch cleanly.

SdpParser — Zero-Copy Field Extraction

Rather than parsing SDP into a full object graph, we scan the raw bytes and extract only the fields we need: o= (origin), a=fingerprint, a=ice-ufrag, a=ice-pwd, and m= lines. We read from a ByteBuffer line by line using index arithmetic, never constructing intermediate substrings for lines we discard.

SessionStore — Lock-Free CAS Transitions

ConcurrentHashMap<String, SdpSession> with VarHandle-based compare-and-swap for state transitions. The transition OFFERING → ANSWERING is only applied if the current state is Offering, preventing double-answer races.

https://github.com/sysdr/discord-flux-p/tree/main/day64/flux-day64

SDP is line-delimited. Each line starts with a single letter and =. We only care about a handful of them. The parser reads raw bytes from the WebSocket payload buffer:

This allocates one ParsedSdp record per session instead of a full Map<String, String> with dozens of entries. Memory footprint drops by roughly 70% versus a generic SDP object model.

If two callers race to set an offer for the same callId, exactly one wins the CAS. The loser gets false and can signal an error immediately instead of silently overwriting.

The WebSocket server dispatches each connection to a Virtual Thread. ICE candidates arrive as additional messages on the same connection. Because Virtual Threads are cheap (~1 KB stack vs ~512 KB for platform threads), a burst of 10,000 trickle ICE messages spawns 10,000 Virtual Threads without exhausting the OS.

Monitor sdp.session.failed_total broken down by failure reason. A spike in ANSWER_TIMEOUT indicates network issues. A spike in CAS_REJECTED means clients are retrying offers — usually a client-side bug or duplicate connection.

Before running anything, confirm these tools are installed:

wscat (optional) lets you send raw WebSocket messages manually during exploration. Install it with npm install -g wscat if you want to poke the server by hand.

Step 1 — Generate the workspace

chmod +x project_setup.sh
./project_setup.sh
cd flux-day64

This creates the full Maven project tree with all Java source files and lifecycle scripts.

Step 2 — Start the server

./start.sh

start.sh compiles the project with Maven, packages it into a JAR, and starts the server in the background. It writes the server process ID to .server.pid.

==> Building flux-day64...
==> Starting SDP Gateway (WS :8080, Dashboard :8081)...
==> Server PID 84312
==> Dashboard: http://localhost:8081/dashboard

Two ports will be open:

• ws://localhost:8080 — WebSocket signaling endpoint

• http://localhost:8081/dashboard — live session dashboard

The unit test suite covers SDP parsing, all state transitions, the fail() spin-CAS loop, and a concurrent race test that verifies exactly one offer wins when 10 Virtual Threads compete simultaneously.

mvn test

Expected output:

[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0

./demo.sh

demo.sh runs LoadTestClient with 20 concurrent call pairs. Each simulated pair goes through the full handshake: OFFER → ANSWER → 3 ICE candidates → ESTABLISH. All 20 pairs run concurrently via Virtual Threads.

Watch the dashboard at http://localhost:8081/dashboard while the demo runs. You will see sessions appear in OFFERING state, transition through ANSWERING, and land on ESTABLISHED within a few hundred milliseconds.

Console output:

[Load] Starting 20 call pairs on localhost:8080
[Load] Done. Success=20 Failed=0
==> Done. Check dashboard: http://localhost:8081/dashboard

./verify.sh

verify.sh calls /api/sessions via curl and checks two things:

==> Verifying session state via API...
    Sessions established : 20
    CAS collisions       : 0
[PASS] At least one session reached ESTABLISHED
[PASS] No CAS collisions (clean state transitions)

To stop the running server without removing build artifacts:

./stop.sh

==> Stopped server PID 84312

To remove build artifacts and log files (does not kill the server):

./cleanup.sh

==> Cleaning build artifacts...
==> Done.

When you want a completely clean slate, run stop.sh first, then cleanup.sh.

Beginner: Add a RINGING state between IDLE and OFFERING. Implement a 30-second ring timeout that transitions the session to FAILED with reason NO_ANSWER.

Intermediate: Replace the String.split("\r?\n") SDP parser with a ByteBuffer-based line scanner that avoids creating a String[] array entirely. Benchmark both with JMH and compare allocation rates.

Expert: Implement SDP munging: before forwarding an OFFER to the callee, rewrite the c= (connection) line to point to your media relay server IP. This is what production TURN servers do to NAT-punch on behalf of clients behind symmetric NAT.