How to Design a Distributed Messaging Queue

Imagine what would happen if Uber tried to calculate your fare, match a driver, estimate the ETA, and ping notifications all in a single synchronous chain before responding to your ride tap. Each step takes a few hundred milliseconds, so running them back-to-back means every request takes seconds.

At a scale of ~100 million requests daily, that blocking setup breaks down completely.

Instead, the initial request simply drops an event onto a queue and immediately sends a response back to the app. Downstream services then pick up that event in parallel and process their piece of the job at their own speed.

If you’re pitching an async setup like this in a system design interview, don’t just stop at “use a queue.” Name the tool (like Kafka or RabbitMQ), specify your delivery guarantees (at-least-once vs. exactly-once), and explain how you’ll prevent duplicate work or lost data using idempotency, retries, and dead-letter queues.

This post covers all of it : from first principles to the internals of how Kafka actually works, with real code and the trade-offs you need to articulate in interviews.

Without a messaging queue, services are coupled at runtime:

Temporal decoupling — producer and consumer don’t need to be available simultaneously. Producer publishes and continues. Consumer processes when ready.

Rate decoupling — producer can publish at 100K msg/sec. Consumer processes at 10K msg/sec. The queue absorbs the difference. Without it, the producer would overwhelm the consumer.

Fault isolation — if Notification Service crashes, orders still process. The notification messages sit in the queue until the service recovers.

Fan-out — one “order.created” event consumed independently by Payment, Inventory, Notification, and Analytics — each with their own offset, their own processing speed, their own retry logic.

Functional requirements:

• Producers publish messages to named topics

• Consumers subscribe to topics and receive messages

• Messages delivered in order within a partition

• Configurable retention (hours to forever)

• Support multiple consumer groups reading same topic independently

• Dead Letter Queue for messages that repeatedly fail

Non-functional requirements:

• High throughput: millions of messages per second

• Low latency: < 10ms publish, < 50ms end-to-end

• Durability: no message lost once acknowledged

• Fault tolerance: broker failure does not cause data loss

• Horizontal scalability: add brokers, add throughput linearly

Scale:

Messages per second: 1 million (write), 10 million (read)
Message size: avg 1KB, max 10MB
Retention: 7 days default, configurable up to forever
Topics: 100,000+
Partitions per topic: 1 to 1,000
Consumer groups: unlimited (each reads same data independently)
Brokers in cluster: 3 to hundreds
Replication factor: 3 (standard)

Apache Kafka avoids using temporary data structures entirely. Instead, it relies on a highly permanent storage structure called a log. A log is simply a sequential data file saved directly on the physical hard drive of the broker server. New incoming data payloads are strictly appended to the very end of this persistent file.

This is the most important insight in distributed messaging:

the queue is not a queue — it’s a log.

Traditional queues (think RabbitMQ or SQS) work like this:

Producer → [msg1, msg2, msg3] → Consumer consumes → message deleted

Message consumed = message gone. Only one consumer can receive each message.

A commit log works differently:

This single design decision enables:

• Multiple consumer groups reading the same topic without interfering

• Message replay — rewind your offset to reprocess historical messages

• Time travel — “give me all messages from last Tuesday at 2pm”

• No lock contention between consumers — each tracks its own offset

A topic is a named stream of messages. Producers publish to topics. Consumers subscribe to topics. Think of a topic like a database table — it has a name, it holds records, it persists data.

Topics in a ride-sharing system:
ride.requests ← new ride bookings
driver.locations ← GPS updates every 5 seconds
payments.completed ← payment confirmation events
notifications.push ← push notification jobs

Kafka topics split into partitions, each handling a subset of messages. Think of it like multiple checkout lanes at a grocery store — instead of one slow line, you have parallel processing lanes. Each partition maintains order within itself while allowing parallel consumption.

Partition key determines assignment:

Why partition key matters:

• Same key → same partition → ordered delivery within that key

• Different keys → different partitions → parallel processing

• Use userId to guarantee all events for one user arrive in order

• Use cityId to parallelize across cities

Each broker is a server in the Kafka cluster. Partitions are distributed across brokers: