Three servers, twelve requests. Send four to each and you are done.
Except requests are not the same size. Three of those twelve are eight times more expensive than the rest, and where they land decides whether the system is balanced or on fire.
imbalance: 1.73× the mean
Counts requests, not work. Three of these twelve requests are eight times more expensive, and whichever server happens to receive them absorbs all of it.
The policies
Round robin. Next server, in turn. Counts requests, not work — and the imbalance above is entirely because of that. Fine when requests are uniform, which is rarer than it sounds.
Least connections. Send to whichever server has fewest in flight. This is a proxy for “least busy” and it is usually the right default. It needs live state, so it belongs in the load balancer rather than in each client.
Two random choices. Sample two servers, pick the lighter. Nearly as good as least-connections with none of the global state, and dramatically better than pure random — the famous “power of two choices” result, which reduces the maximum load from to .
Weighted. Bigger machines get proportionally more. Composes with any of the above.
Consistent hash. Route by a key so the same user or cache key lands on the same server. See consistent hashing — this is how you get cache locality, at the cost of even distribution.
Layer 4 or layer 7
L4 balances TCP connections. It sees addresses and ports, nothing else. Very fast, works with any protocol, and cannot route by path or read a header.
L7 terminates HTTP and sees the request. Route /api separately from
/static, retry a failed request on another server, do TLS termination and
compression. Slower, and far more useful for an HTTP application.
Most architectures use both: L4 at the edge for raw throughput, L7 behind it for routing.
Health checks decide everything
A balancer is only as good as its idea of which servers are alive.
Passive — notice failing responses and back off. Free, and it learns only by sending real users to a broken server.
Active — poll /health on a timer. Detects failure before a user does, and
costs a request per server per interval.
Sticky sessions, and why to avoid them
Pinning a user to one server lets you keep session state in memory. It also means that server’s failure logs those users out, that scaling down drops sessions, and that load cannot rebalance.
Better: keep sessions in Redis or a signed cookie, and let any server take any request. Statelessness is what makes everything else in this article work.
Above the load balancer
One balancer is a single point of failure. The layers above it:
DNS returning several A records — free, and clients cache aggressively, so removing a bad address takes minutes.
Anycast — the same IP announced from many locations; BGP routes each user to the nearest. This is how CDNs and public DNS resolvers work, and it fails over in seconds without the client knowing.
Client-side balancing — the client holds the server list and picks. Removes a hop entirely; used inside service meshes, where a sidecar proxy does it per pod.