Load Balancing in System Design
Learn how load balancers route traffic, detect unhealthy instances, drain connections, and support scalable, resilient services.
What you'll learn
Build your mental model
- How routing algorithms, health checks, and connection draining interact.
- Why session state and uneven requests complicate simple round robin.
- How to discuss traffic routing and failure behavior in an interview.
A load balancer is a traffic decision point, not simply a round-robin box. It chooses which request should reach which healthy capacity while preserving the connection, locality, affinity, and failure behavior the application actually needs.
Start with the decision canvas, then use the guide to test the model against scale and failure.
Read the conditions before the components.
ATOFF architecture reasoning canvas
Load Balancing in System Design: the mental model
Short answer
Load balancing is a routing policy that sends each request to suitable healthy capacity while respecting connection and locality constraints.
Proof signal
Healthy capacity completes the promised work; it does not merely answer a ping.
01 Decision checkpoint
Core model
Round robin suits equal short work, while least-loaded, locality-aware, or affinity-aware routing fits uneven work and stateful connections.
02 Decision checkpoint
Pressure test
A port can accept traffic while its database pool is exhausted, so health checks must reflect whether the instance can serve the promised request.
03 Decision checkpoint
Design move
Remove capacity gradually, stop assigning new work, and let active requests or sockets finish within a bounded deadline.
Say it in the interview
I would pick the routing algorithm from workload shape and describe health checks, draining, and behavior when capacity falls below demand.
- 1
Clients
Load balancer
- 2
Load balancer
Healthy instance A
- 3
Healthy instance A
Healthy instance B
- 4
Healthy instance B
Draining instance
- 5
Draining instance
- Clients flows to Load balancer.
- Load balancer flows to Healthy instance A.
- Healthy instance A flows to Healthy instance B.
- Healthy instance B flows to Draining instance.
From model to real-world behavior
Routing is a policy decision
Round robin works when requests cost about the same and connections are short. Least-connections can help long-lived work. Weighted routing enables gradual rollout. Consistent-hash routing can keep a user's cache or session near the same backend. Each algorithm assumes something about workload shape.
Start by asking whether requests are interchangeable. If one request streams a file for minutes and another reads a profile in milliseconds, request count is a poor proxy for resource use.
Healthy means able to serve the relevant work
A process that answers a ping may still be unable to reach its database or may be overloaded enough to violate user latency. Health checks should distinguish liveness from readiness, use conservative failure thresholds, and avoid sending probe traffic that itself becomes a source of pressure.
During deploys, connection draining matters as much as adding capacity. Stop new requests before terminating an instance, allow bounded in-flight work to finish, and make the client retry path safe when a connection closes.
State and locality shape the architecture
The easiest systems load-balance stateless handlers and put state in a shared durable store. Sticky sessions can be a transitional choice, but they skew load and make a failed instance more disruptive. If locality is valuable, use it consciously for a cache or streaming session rather than accidentally coupling user state to a machine.
A global load balancer introduces a second decision: which region should own the request? Latency, data residency, regional health, and failover policy all matter more than a generic nearest-region rule.
Interview and staff-level signal
A solid interview answer names the routing policy, health signal, and session strategy. A stronger answer adds overload behavior: where queues form, how the edge sheds traffic, and how a rollout avoids sending all traffic to a fresh version.
Staff-level designs treat traffic policy as a controllable product surface. They include observability for regional skew, per-route latency, health flapping, and saturation, plus a tested manual override for the incident commander.
Keep this with you
Key takeaways
- Choose a routing algorithm based on workload behavior, not habit.
- Readiness, draining, and overload handling determine whether routing is actually resilient.
- Keep state out of individual instances unless affinity is intentional and bounded.
Practice aloud
Interview questions to explore
- 1.Why is round robin insufficient for this workload?
- 2.How do you detect an instance that is alive but unable to serve correctly?
- 3.What happens to in-flight connections during a rollout?
Common follow-ups
Frequently asked questions
Is an API gateway a load balancer?
They can both route traffic, but an API gateway usually owns application-edge policy such as authentication, rate limits, and API composition. A load balancer focuses on distributing traffic across capacity.
Do sticky sessions prevent scaling?
Not automatically, but they reduce flexibility and make failures harder to absorb. Stateless application instances are usually easier to scale and recover.
Need the broader preparation context? Go back to Interview Preparation for behavioral readiness, question practice, and the larger AceTheOffer preparation framework.