Rate Limiting in System Design
Learn how to design rate limits that balance fairness, burst handling, distributed counters, and safe failure behavior for APIs.
What you'll learn
Build your mental model
- How fixed windows, sliding windows, and token buckets change user experience.
- Why identity, scope, and distributed atomicity matter more than the counter alone.
- How to describe a rate limiter's failure posture in an interview.
Rate limiting is a fairness and protection policy. It decides how much of a scarce resource one actor can consume over time, what a legitimate burst looks like, and how the system responds without letting noisy or abusive traffic degrade everyone else's experience.
Start with the decision canvas, then use the guide to test the model against scale and failure.
Read the conditions before the components.
ATOFF architecture reasoning canvas
Rate Limiting in System Design: the mental model
Short answer
Rate limiting protects shared capacity by defining who may consume what amount of work, over which window, and with which burst allowance.
Proof signal
Rejecting low-value work early protects the requests that must still succeed.
01 Decision checkpoint
Core model
Prefer tenant, API key, user, route, and cost dimensions over IP alone so fairness follows the product's actual ownership model.
02 Decision checkpoint
Pressure test
A token bucket can admit a short useful spike; a fixed window can create accidental cliffs at its boundary.
03 Decision checkpoint
Design move
Return a clear retry signal, preserve essential traffic, and choose whether a limiter-store outage should fail open or fail closed by endpoint risk.
Say it in the interview
I would define the limiting identity and budget first, then choose a distributed counter design and a safe response for counter-store failure.
- 1
Caller
Identity & route
- 2
Identity & route
Rate-limit state
- 3
Rate-limit state
Allow or reject
- 4
Allow or reject
Protected service
- 5
Protected service
- Caller flows to Identity & route.
- Identity & route flows to Rate-limit state.
- Rate-limit state flows to Allow or reject.
- Allow or reject flows to Protected service.
From model to real-world behavior
Start with the policy before the algorithm
A rate limit needs an actor, a resource, a time window, and an intended fairness rule. A per-IP limit can be useful for anonymous abuse but unfair behind shared networks. A per-user limit protects an account API but may not protect a tenant with thousands of users. Limits often need several dimensions.
Decide whether the goal is abuse prevention, cost control, partner tiering, downstream protection, or fairness. The answer determines whether a request should be delayed, rejected, queued, or allowed with lower priority.
Choose an algorithm for the burst behavior you want
A fixed window is easy but allows a caller to spend a full allowance on either side of a boundary. A sliding window smooths that edge at more bookkeeping cost. A token bucket refills steadily and permits a bounded burst, which makes it a common fit for interactive APIs.
The algorithm is only correct if its state is updated atomically at the chosen scope. In a distributed service, a local in-memory counter can overshoot when traffic is balanced across instances; a shared store or partitioned ownership may be needed for stricter limits.
Decide what happens when the limiter is unavailable
Failing open preserves availability but exposes a sensitive dependency to a traffic surge. Failing closed protects the dependency but can block healthy users. The right choice differs for a login endpoint, an internal read API, and a paid customer workflow. It should be explicit and observable.
Return actionable responses: a meaningful status, a retry hint when safe, and headers that help cooperative clients self-regulate. Hidden throttling merely shifts the queue into clients and makes demand harder to see.
Staff-level insight: rate limits are capacity contracts
At larger organizations, rate limits become contracts between platform and product teams. Version policies, audit exceptions, measure rejected work by tenant and route, and coordinate limit increases with capacity plans. The goal is to prevent a local product decision from consuming shared reliability headroom.
Include a safe override path for incident response, but make it narrow, time-bound, and visible. Unlimited access should be an exception with a cost owner, not the easiest way to make an alert disappear.
Keep this with you
Key takeaways
- Rate limiting encodes a fairness and protection policy before it becomes an algorithm.
- Choose state scope and atomicity based on how precise the limit must be.
- Failure posture and client feedback are part of a complete limiter design.
Practice aloud
Interview questions to explore
- 1.What identity key should this limit use, and why?
- 2.How much burst should a legitimate client be allowed?
- 3.Would you fail open or closed if the counter store is unavailable?
Common follow-ups
Frequently asked questions
Is throttling the same as rate limiting?
Rate limiting enforces a policy on allowed work. Throttling usually means intentionally slowing accepted work. Systems may use both depending on whether they need rejection or smoothing.
Can a cache implement a distributed rate limiter?
A fast shared store can hold counters if it offers the atomic operations the algorithm needs. Its availability and partitioning behavior then become part of the limiter's correctness and failure design.
Need the broader preparation context? Go back to Interview Preparation for behavioral readiness, question practice, and the larger AceTheOffer preparation framework.