Lesson 56: Handling Failures

Module 4: The Matching Engine (Processor API)

Lesson 55 assumed a driver was always sitting in the K-Ring. He isn’t. At 2am in a low-density zone, or during a surge event that’s already pulled every nearby driver into other trips, MatchEmitterProcessor runs its gridDisk() lookup and gets back nothing. What you do in that moment is the difference between a rider who sees “Finding your driver...” and one who sees a frozen app. This lesson is about the failure path: retrying without blocking, widening the search radius without brute-forcing the whole map, and telling the frontend the truth when there’s genuinely no one available.

The lesson-55 processor already has the bug baked in — it just fails silently:

The instinctive “fix” a standard developer reaches for is worse:

This fails in three distinct ways, and all three get worse together under load, not independently.

Failure 1: Thread.sleep() inside process() is a StreamThread freeze. Kafka Streams calls process() synchronously from the poll loop. A one-second sleep doesn’t delay one rider — it stalls every partition owned by that StreamThread‘s TaskManager for a full second, per retry, per failed match. At 3,000 events/sec with even a 5% no-match rate during a demand spike, that’s 150 events/sec triggering multi-second stalls. max.poll.interval.ms gets blown past, the consumer group coordinator declares the instance dead, and you get a rebalance — which itself pauses processing for every task on that node, including the 95% of requests that were matching fine. A no-match problem becomes a whole-partition outage.

Failure 2: Unbounded, unsynchronized ring widening is a thundering herd. Every failed request independently grows its own ring and retries on its own clock. During a surge, thousands of riders fail their initial K-Ring search within the same second (because the underlying cause — driver scarcity — is shared). If every one of them retries after exactly 1000ms, you get a synchronized thundering herd hitting RocksDB at the same wall-clock instant, every second, for as long as the shortage lasts. You’ve turned a supply problem into a self-inflicted load spike.

Failure 3: The blocking HTTP call to notify the frontend repeats the exact mistake Lesson 55 fixed for the success path. There’s no reason the failure path should get a pass on the blocking-I/O rule just because it’s the unhappy path. It’s often the more likely path to be blocking under exactly the load conditions where you can least afford it.

We treat “no match” as a first-class state, not an exception. Failed requests move into a persistent retry queue backed by RocksDB, driven entirely by a Punctuator — never by a loop inside process(). The pattern: