Design a Rate Limiter
Practice a distributed rate limiter with tenant fairness, burst control, atomic counters, policy propagation, failure modes, and observability.
What you'll learn
Run the architecture drill
- How to model rate-limit keys, policies, and burst allowance.
- How to make a distributed permit decision safely and quickly.
- How to design for configuration changes and partial counter-store failure.
Designing a rate limiter is an exercise in translating a capacity policy into an efficient, distributed decision. The algorithm matters, but so do the identity hierarchy, regional behavior, configuration propagation, counter-store failures, and the customer experience of a rejected request.
Start with the architecture drill, then follow the request path through scale, failure, and a defensible interview answer.
Make your assumptions and decision path easy to inspect.
ATOFF architecture reasoning canvas
Design a Rate Limiter: the architecture drill
Short answer
A distributed rate limiter should make a fast, explainable admission decision that reflects product hierarchy and remains safe when shared counters are impaired.
10× evolution
Partition counter keys and make policy propagation observable by region.
01 Decision checkpoint
Design brief
Evaluate global, tenant, user, route, and cost budgets in a clear order so one expensive API cannot consume a customer's entire allowance invisibly.
02 Decision checkpoint
First request path
Use an atomic token or counter update close to the request path; separate policy distribution from per-request decision storage.
03 Decision checkpoint
Failure drill
Fail closed for sensitive or costly operations and fail open with local safeguards for essential reads, while emitting a visible operational signal.
Say it in the interview
I would define the identity tree and algorithm, then explain atomic distributed enforcement, configuration rollout, and limiter-store outage behavior.
- 1
Client request
request: API edge identity + route
- 2
API edge identity + route
consume token: Atomic token bucket state
- 3
Policy cache + rollout config
limit + burst: API edge identity + route
- 4
Atomic token bucket state
allow: Protected service · deny: 429 + retry guidance · record outcome: Limit decision telemetry
- 5
Protected service
- 6
429 + retry guidance
- 7
Limit decision telemetry
- Client request flows to API edge identity + route via request.
- Policy cache + rollout config flows to API edge identity + route via limit + burst.
- API edge identity + route flows to Atomic token bucket state via consume token.
- Atomic token bucket state flows to Protected service via allow.
- Atomic token bucket state flows to 429 + retry guidance via deny.
- Atomic token bucket state flows to Limit decision telemetry via record outcome.
Walk the design under pressure
Clarify the fairness contract
Ask whether the limit is per IP, API key, user, tenant, route, or a hierarchy of those dimensions. A public login endpoint may need IP and account protections; a paid API may need a tenant plan, per-route cost, and a short burst allowance. The key determines both fairness and cardinality.
Define the response behavior too: should the client receive an immediate rejection, a retry time, or a lower-priority queue? The user experience should match the purpose of the limit rather than hiding capacity pressure behind random timeouts.
Use an algorithm whose state can be updated atomically
A token bucket is often a good starting point because it allows a bounded burst and refills predictably. Store the current token state and last refill time as one atomic operation. A fixed window is simpler but creates boundary bursts; a sliding approach is fairer but more expensive to compute or store.
For strict global limits, requests need to converge on authoritative counter state. For lower-risk protection, regional local limits may be acceptable and faster. State the overshoot tolerance instead of pretending every distributed counter is globally exact.
Policy propagation and failure posture are first-class
Policies change when a customer upgrades, an incident begins, or a route becomes expensive. Cache configurations with a version and safe TTL, then make urgent overrides visible and bounded. Counters should not be reset accidentally when policy changes unless the product rule says they should.
If the counter store is unavailable, decide per route whether to fail open, fail closed, or use a conservative local fallback. Log that degraded mode; otherwise a limiter outage becomes either an invisible abuse window or a mysterious product outage.
Staff-level insight: turn limits into capacity governance
A rate limiter becomes a platform contract. Build reporting by tenant, route, outcome, and policy version; review exceptions; and tie large limit increases to capacity planning. That keeps local product launches from consuming shared headroom without a coordinated decision.
In an interview, distinguish the free educational algorithm from the production design: atomicity, observability, configuration safety, regional tradeoffs, and incident overrides are where a simple limiter becomes a trusted platform capability.
Keep this with you
Key takeaways
- Rate limiting begins with a fairness policy and identity hierarchy.
- Distributed atomicity and tolerated overshoot determine whether a limiter is actually enforcing its rule.
- Configuration changes and counter-store failures need explicit, observable behavior.
Practice aloud
Interview questions to explore
- 1.What exact key defines a fair limit for this endpoint?
- 2.How precise must a global limit be during a regional network partition?
- 3.How do policy changes reach every edge safely?
Common follow-ups
Frequently asked questions
Why not use an in-memory counter in every API server?
It is fast but each instance sees only its own traffic. A client can exceed a global limit by being routed across instances unless the design accepts that approximation or coordinates shared state.
What is a token bucket?
It is a limiter model with tokens that refill at a steady rate up to a capacity. A request spends tokens, allowing controlled bursts while enforcing a sustained rate.
Already an Elite member? Open the complete walkthrough.
Need the broader preparation context? Go back to Interview Preparation for behavioral readiness, question practice, and the larger AceTheOffer preparation framework.