We’ve already covered Rate Limiting in detail, including the implementation of the most common rate-limiting algorithms. If you’d like to dive deeper, you can read the complete series of articles below:
A few years ago, a team I know deployed a rate limiter in front of their payments API. The configuration looked straightforward: 100 requests per minute per client. Everything was tested, dashboards looked healthy, and the deployment went smoothly.
A few weeks later, one client exhausted their entire monthly quota with a downstream payment provider in just a few seconds.
Nothing was misconfigured. The rate limiter worked exactly as designed.
The problem was that it enforced 100 requests per minute, while allowing those 100 requests to arrive almost at the same time. The limit was respected, but the sudden burst was enough to overwhelm the downstream system.
The rate limiter wasn’t broken. It simply wasn’t protecting against the kind of traffic that actually caused the problem.
Most engineers think of rate limiting as a solved problem. Choose an algorithm, connect it to Redis, set a limit, and you’re done.
But a rate limiter that works in testing isn’t necessarily one that works under real production traffic.
Most failures don’t happen because the limiter stops working. They happen because it allows the very traffic pattern it was supposed to prevent—usually during traffic spikes, retry storms, or unexpected bursts.
If you’re building APIs, gateways, or multi-tenant systems, choosing the right rate-limiting strategy is critical. The wrong one can give you a false sense of protection.
Here’s the key idea behind this post:
A rate limiter doesn’t just limit requests—it controls how traffic reaches your system.
Consider three clients, each limited to 100 requests per minute.
Client A sends one request every 600 milliseconds.
Client B sends all 100 requests in the first second, then stays idle.
Client C sends 50 requests just before the minute ends and another 50 immediately after the next minute begins.
All three follow the same limit.
But your database experiences them very differently.
Clients B and C create sudden bursts of traffic, while Client A generates a smooth, predictable load.
Think of a highway entrance with a traffic light that lets one car through every two seconds. If cars arrive evenly, traffic flows smoothly. But if several cars are released together, congestion can still happen—even though the traffic light’s rules weren’t broken.
That’s the key lesson.
Most rate limiters are judged by how many requests they allow. What really matters is how those requests arrive. Different algorithms handle traffic bursts in different ways, and each comes with its own trade-offs. Understanding those trade-offs is what separates a working rate limiter from one that truly protects your system.
A junior engineer sees rate limiting as a technical problem: pick a well-known algorithm, implement it, and move on.
A senior engineer goes a step further. They ask whether it’s thread-safe, whether it works during Redis failover, and whether it can handle production scale.
An architect starts with a different question:
What am I trying to protect, and what does failure actually look like?
That question changes everything.
If you’re protecting a database connection pool, the biggest risk is too many requests arriving at the same time. A short burst of traffic can exhaust the pool, even if your average requests per minute look perfectly healthy.
If you’re protecting a public API from abuse, the problem is different. Here, the concern is how many requests arrive over a longer period, not a brief spike.
If you’re protecting a third-party billing API with a strict monthly quota, then neither bursts nor concurrency matter as much. What matters is the total number of requests sent over time.
All three are called rate limiting, but they’re solving completely different problems.
That’s where architects think differently. They don’t choose a solution because it matches the problem’s name. They choose it because it matches the failure they’re trying to prevent.
Junior engineers focus on the implementation.
Senior engineers focus on making the implementation reliable.
Architects first make sure they’re solving the right problem.
That’s why rate limiting isn’t just about choosing the right algorithm. It’s about understanding what you’re protecting before you decide how to protect it.
Before we dive into the failure modes in Part 2, let’s understand the three main rate-limiting algorithms. You don’t need to know their implementation yet—just how each one controls traffic.
Imagine a bucket that can hold 100 tokens. New tokens are added at a steady rate, for example 10 tokens every second, until the bucket is full.
Each incoming request consumes one token. If there are no tokens left, the request is rejected.
The key idea is that Token Bucket allows short bursts of traffic. If a client has been idle for some time, the bucket fills up. That client can then send a large burst of requests immediately, as long as enough tokens are available.
A Sliding Window doesn’t divide time into fixed intervals like “this minute” or “this hour.” Instead, it continuously looks at a rolling period, such as the last 60 seconds.
This avoids the traffic spikes that can happen at the boundary of fixed windows.
The trade-off is that the algorithm needs to keep track of request timestamps rather than just a simple counter. At large scale, storing and processing those timestamps increases memory usage—something we’ll explore later in this chapter.
Think of requests being poured into a bucket. The bucket processes them at a constant rate, no matter how quickly they arrive.
If requests arrive faster than they can be processed, they wait in the queue. Once the bucket is full, any additional requests are rejected.
The key idea is that Leaky Bucket smooths out traffic. Even if 50 requests arrive at once, they are processed one by one at a fixed rate.
This creates predictable traffic for downstream systems, but it also increases latency. A perfectly valid burst of requests may not be rejected—it may simply wait long enough for the client to time out.
Here’s roughly where each algorithm sits in a typical request path:
This diagram looks the same no matter which rate-limiting algorithm is used. That's the catch. The diagram only shows where the decision is made—not when requests are allowed or denied. Two systems can have the exact same architecture but behave very differently when traffic suddenly spikes. In Part 2, we'll look inside the rate limiter to see why.
To understand the basic idea, let's look at a simplified C# example. This isn't a complete implementation—it's just enough to show how a Token Bucket works.
Notice what's missing: the code doesn't track when a client last sent a burst of requests. It only refills tokens based on time. That simple design choice is what allows bursts of traffic, as we'll see in Part 2.
Companies like Stripe and Cloudflare use different rate-limiting algorithms based on the traffic they need to manage. Public documentation shows that there is no single "best" algorithm. The choice depends on what they're trying to protect—whether it's API stability, backend capacity, or abuse prevention.
That's exactly the architect's mindset: start by understanding the problem before choosing the solution.
Mistake: Treating “rate limiter” as a single interchangeable component — swapping token bucket for sliding window without re-checking whether the failure mode that mattered for the old choice still applies to the new one.
Why it breaks: Each algorithm is designed for a different traffic pattern. Replacing one with another without understanding why it was chosen can create new problems.
Better alternative: Before choosing an algorithm, define what you're protecting. Is the concern sudden bursts, sustained traffic, concurrent connections, or request cost? Let that answer guide your decision.
A rate limiter controls how traffic reaches your system, not just how many requests it receives.
Different algorithms allow different traffic patterns, even with the same rate limit.
Token Bucket allows short bursts.
Sliding Window smooths traffic but requires more memory.
Leaky Bucket delivers a steady request rate but may increase latency.
The most important question isn’t “Which algorithm is best?” It’s “What am I trying to protect?”
In Part 2, we’ll see how each algorithm behaves under real production traffic and where it can fail.
This publication is supported by readers who want to build systems that survive real production traffic—not just pass a load test. If you found this article useful, consider subscribing to unlock Parts 2 and 3, where we'll explore how these algorithms behave under load and how to choose the right one for your architecture.
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.