How Load Balancing Actually Works

Most people first meet load balancing as a tidy diagram: one box on the left, several servers on the right, arrows fanning outward. The box is supposed to spread traffic evenly and somehow make the service fast and reliable.

That picture is useful in the way that saying a railway station "moves trains around" is useful. It is true, and it skips the machinery that decides whether the system holds together when a rack loses power, when one backend gets slow rather than dead, when HTTP/2 collapses thousands of browser requests onto a few long-lived connections, when a deployment drains old instances while new ones are still warming their caches, or when global traffic needs to move away from an entire city.

A real load balancer is usually not one thing. It is a stack of steering decisions across different layers:

• DNS or anycast decides which edge or region the client reaches first
• an L4 balancer decides which frontend host should own the flow
• an L7 proxy decides which upstream service instance should receive the request
• health systems and outlier detection decide which backends are temporarily not trustworthy
• retry and draining rules decide what happens when a backend is alive enough to answer but unhealthy enough to hurt users

Those decisions do not operate on the same data and they do not fail in the same way. That distinction is the heart of the subject.

This article walks through the mechanics from the first SYN to the final backend choice. We will look at five-tuple hashing, ECMP, Maglev-style consistent flow steering, L7 algorithms such as least requests and power of two choices, stickiness, health checks, connection draining, global traffic steering, and the failure modes that make load balancing look easy in slides and complicated in production.

Load balancing is a chain of steering layers, not one decision

The first mistake people make is asking, "Which algorithm does the load balancer use?" as if one answer covers the whole path.

In a modern service, different parts of the system balance different things:

Client in Athens
  -> DNS or anycast chooses an edge or region
  -> L4 balancer chooses a frontend host by flow hash
  -> frontend terminates TLS and parses HTTP
  -> L7 router chooses a backend instance
  -> backend pool may call other internal services through more balancers

Even a modest SaaS deployment often has at least two balancing layers:

1. a public entry layer that owns IPs and TCP or UDP flows
2. an application proxy layer that understands HTTP, gRPC, or WebSocket semantics

Large operators add more:

• regional steering before the traffic even reaches a data centre
• rack-aware or zone-aware balancing inside the service mesh
• storage or cache sharding behind the application tier

Each layer optimises for a different unit of work.

At L4, the unit is usually a flow. The balancer sees source and destination IPs, source and destination ports, and protocol. It can forward packets quickly because it does not need to parse HTTP headers or gRPC metadata. It is good at moving lots of traffic.

At L7, the unit is usually a request. The balancer can look at hostnames, paths, cookies, methods, headers, or gRPC service names. That makes smarter routing possible, but it means terminating connections, parsing bytes, and holding more state.

The distinction matters because one TCP connection may contain many requests. Under HTTP/1.1 without much keep-alive reuse, the difference is small. Under HTTP/2 and HTTP/3, it is enormous. A browser might open one connection to the edge and multiplex dozens of streams over it. An L4 balancer never sees those streams. An L7 proxy does.

That is why "balanced connections" and "balanced requests" are not the same claim.

The fast path usually starts with L4 flow hashing

Suppose a client in Lisbon opens a TCP connection to api.example.eu. DNS or anycast has already landed the traffic in a Frankfurt PoP. The first public box that handles the packet is often an L4 balancer or a fleet of L4 balancers.

Its basic job is simple: make sure every packet of the same flow reaches the same next hop.

A typical flow key is the five tuple:

source IP
source port
destination IP
destination port
protocol

The balancer hashes that tuple and uses the result to pick a backend host or a slot in a lookup table. The important word is consistently. If the SYN goes to frontend fe-03, the SYN-ACK and all later packets for that connection must also land on fe-03, otherwise TCP state breaks immediately.

A toy version looks like this:

hash = H(src_ip, src_port, dst_ip, dst_port, proto)
backend = table[hash % N]

Real systems are more careful than mod N, because simple modulo has an ugly property: if N changes, almost every flow remaps. Add one frontend and a large fraction of active connections jump elsewhere. For stateful protocols that is a disaster.

This is why production balancers use techniques such as:

• consistent hashing rings
• rendezvous hashing
• fixed lookup tables with controlled reprogramming
• Maglev-style permutation tables

Google's Maglev paper is famous because it describes a practical way to build a large fixed lookup table that keeps remapping low when the backend set changes. The table might have 65,537 slots, each assigned to one backend according to a deterministic fill algorithm. A packet hash lands in one slot, and the slot says which backend owns the flow. If one backend disappears, only the slots that belonged to it need to move.

That property matters operationally. If a frontend host dies, you want only its share of flows to move, not the entire service's traffic pattern.

A simplified table view:

slot 0     -> fe-01
slot 1     -> fe-07
slot 2     -> fe-03
slot 3     -> fe-03
slot 4     -> fe-12
...
slot 65536 -> fe-02

This looks boring until a host fails. Then the table becomes the difference between a local disturbance and a global reconnection storm.

ECMP is load balancing too

Inside data centres and backbone networks, Equal-Cost Multi-Path routing, ECMP, performs a similar function at the routing layer. If several next hops have the same cost, the router hashes the flow and picks one path. That spreads traffic across spine links without reordering packets within each flow.

This is an important connection between network routing and service load balancing. The same principle appears at several layers:

• switches spread flows across links
• L4 balancers spread flows across frontend hosts
• proxies spread requests across service instances

If the hashes are poorly chosen, elephants collide on the same path and the network looks randomly overloaded. If the hash inputs are stable and well distributed, traffic spreads far more smoothly.

Direct Server Return removes the balancer from the heavy side of the flow

A common scaling trick at L4 is Direct Server Return, DSR. The load balancer handles the ingress side, but the server replies directly to the client instead of sending response traffic back through the balancer.

That matters because many workloads are asymmetric. A user request may be 1 KB. The response may be 200 KB, 2 MB, or a video segment. If the balancer must carry both directions, its egress requirement becomes enormous.

In DSR, the packet path looks like this:

client -> L4 balancer -> selected server
server -> client

The trick requires careful address handling. The server must be able to accept packets that arrived via the virtual IP, yet emit replies that still make sense to the client. Linux IPVS, hardware ADCs, and hyperscale platforms have supported variations of this for years.

DSR is a reminder that load balancing is often about bandwidth placement as much as algorithm choice. The fastest scheduler in the world will not save a design that forces all response bytes through the wrong box.

L7 balancing operates on requests, which changes the tradeoffs completely

Once traffic reaches an application proxy, the balancer can make richer decisions because it has more context. It can terminate TLS, inspect Host, parse paths, understand methods, and sometimes read service metadata such as gRPC authority and method names.

That opens the door to routing rules like:

• api.example.eu/v1/payments/* goes to the payments service
• static.example.eu/* goes to the cache tier
• requests with tenant cookie tenant=eu-west stay in that tenant's pool
• gRPC method inventory.Stock/Reserve goes to inventory backends only

It also opens the door to smarter balancing algorithms, because the proxy can track request-level metrics rather than only connection existence.

Round robin is simple and often fine

The simplest algorithm is round robin:

request 1 -> backend A
request 2 -> backend B
request 3 -> backend C
request 4 -> backend A

Round robin is attractive because it needs almost no state. If all backends are identical and requests cost roughly the same, it works well enough.

That "if" is doing a lot of work.

In real systems, requests differ wildly:

• one request fetches a 2 KB profile document
• one request triggers a 400 ms database query
• one request streams 50 MB of export data
• one request upgrades to a WebSocket and stays open for hours

Round robin does not know any of that. It only knows the order.

Least connections and least requests track live occupancy

A better approximation is least connections or least requests. The proxy sends the next request to the backend with the smallest active count.

That helps when some requests take longer than others. A backend already holding 400 live requests probably deserves less new work than one holding 180.

The catch is that counting alone can still lie.

Example:

backend-a: 40 live requests, median latency 18 ms
backend-b: 35 live requests, median latency 250 ms

Pure least requests will prefer backend-b, because 35 is smaller than 40, even though backend-b is already slow and likely overloaded. Occupancy is useful, but incomplete.

Power of two choices gets most of the win without global coordination

One of the best practical load balancing ideas is the power of two choices. Instead of scanning the whole pool, the balancer picks two random healthy candidates and chooses the better one by some local metric, often active requests.

That sounds almost too simple. It works remarkably well.

Why? Because pure random has bad tail behaviour. With enough requests, some servers get unlucky streaks and become hot. Checking two candidates collapses that tail sharply without requiring the balancer to maintain perfectly synchronised global state.

A toy example:

random pick candidates: backend-b and backend-f
active requests:
  backend-b = 92
  backend-f = 37
choice -> backend-f

This is cheap, decentralised, and good enough for many large systems. Envoy, nginx variants, and service-mesh components have all leaned on this family of ideas.

Latency-aware scoring reacts to slow backends better than counts do

The next step is to incorporate observed latency. If a backend's response times rise, it should receive less new work before it becomes completely broken.

A common pattern is an exponentially weighted moving average, EWMA. The balancer tracks recent latency per backend and scores lower-latency backends more favourably.

Conceptually:

score = ewma_rtt * penalty_for_active_requests
pick backend with lowest score

This is better than least requests when a server is not dead, just bad. Maybe the JVM is in a long GC pause. Maybe one node lost local cache warmth after a restart. Maybe a database replica behind that backend is lagging. The server still answers health checks, so a binary healthy or unhealthy model is too blunt. EWMA gives the scheduler a way to shift traffic away before users feel the full blast.

The difficulty is avoiding feedback loops. If latency spikes briefly, overreacting can send a surge elsewhere, creating a second hot spot. Good implementations damp the signal rather than chasing every blip.

Weighted balancing handles intentionally unequal capacity

Not every pool is homogeneous. You may have:

• eight modern nodes and two smaller emergency nodes
• one rack in maintenance mode with CPU limits applied
• canary instances that should receive 5 percent of traffic
• mixed ARM and x86 fleets with different performance per core

Weighted round robin or weighted least requests lets the operator express that asymmetry.

Example HAProxy snippet:

backend api_pool
    balance leastconn
    server api-a 10.0.10.11:8443 weight 100 check
    server api-b 10.0.10.12:8443 weight 100 check
    server api-canary 10.0.10.99:8443 weight 5 check

Here the canary exists in the same pool but receives only a thin slice of traffic. That is not marketing "progressive delivery" language. It is simply weighted balancing doing exactly what it says.

Stickiness and state change what "balanced" even means

Pure balancing assumes any backend can handle any request. That is only true when state is either absent or externalised.

Many systems keep some state local to a backend:

• in-memory session objects
• local cache entries
• WebSocket connection state
• model weights or warm compiled artefacts
• shard-specific ownership of some key range

In these cases, the balancer often needs some form of affinity, also called stickiness.

Session affinity trades fairness for locality

Suppose a shopping basket lives in backend memory. If a user's first request hits app-03, later requests for that user either need to keep hitting app-03 or the state has to move elsewhere. Moving it on every request is silly, so operators often choose affinity.

The sticky key might be:

• a cookie inserted by the proxy
• a session ID hashed by the balancer
• the client IP, though that is increasingly unreliable and unfair
• a tenant ID
• a consistent hash on a URL path or object key

Example idea:

hash(cookie: SESSIONID=9f31...) -> backend 7

The benefit is locality. The cost is imbalance. If one tenant is noisy or one client opens many long-lived connections, the sticky backend can become much hotter than the rest.

This is why IP-based stickiness is frequently a bad default. One large enterprise customer behind a single egress NAT in Milan can make thousands of unrelated users appear to come from one source IP. If stickiness keys on client IP, those users collapse onto the same backend.

Consistent hashing helps when the key matters more than perfect fairness

Cache clusters are a classic example. If request key product:48192 lands on node A today and node A tomorrow, the second request gets a warm cache hit. If membership changes and every key reshuffles, the cache turns cold everywhere.

Consistent hashing keeps remapping modest when nodes are added or removed. With virtual nodes or weighted slots, the ring can also absorb uneven capacity.

A ring sketch:

0 ---------------------------------------------------- 2^32
   A   A   B   C   C   D   A   B   D   C
 
hash(product:48192) -> lands between C and D -> node D

The ring is not limited to caches. It appears in API gateways, queue consumers, rate-limiting shards, and pub-sub partition ownership.

The downside is that consistent hashing optimises for key locality, not instantaneous fairness. If one hot key dominates traffic, the lucky owner of that key still melts first.

HTTP/2 and HTTP/3 make connection-level balancing more deceptive

Under HTTP/2 or HTTP/3, a client can send many logical requests over one connection. If your public edge balances only at connection setup time, request distribution downstream may look skewed.

A browser might open six TCP connections under HTTP/1.1, but only one or two HTTP/2 connections. If one of those lands on a busy frontend, a large share of the user's request volume follows it. This is one reason L7 request-aware balancing became so important. Connection counts alone stopped being a reliable proxy for load.

gRPC made this even more obvious. Long-lived multiplexed channels are efficient, but they hold onto whichever upstream was chosen unless the client or proxy actively rebalances. Many production incidents have come from gRPC clients pinning too much work to a subset of endpoints after scale-out events.

Health checking is not about dead or alive. It is about trust

A backend can fail in several ways:

• hard down, no TCP accept
• process alive, but returns 500s
• accepts connections, but latency is catastrophic
• only one dependency is failing, so a subset of requests break
• draining for deployment, so it should finish old work but avoid new work

If the balancer only distinguishes "reachable" from "not reachable", it reacts too late.

Active checks probe the service explicitly

An active health check is a probe sent by the balancer on a schedule. It might be:

• TCP connect
• HTTP GET /healthz
• HTTPS request with expected status and body
• gRPC health checking call

Example:

GET /readyz HTTP/1.1
Host: api.internal
 
HTTP/1.1 200 OK
ready=true
db=ok
queue_lag_ms=12

A good readiness endpoint does not merely prove the process is running. It proves the service is willing to take new work. That usually means critical dependencies are in a sane state.

Passive checks watch real traffic for signs of trouble

Passive checks use observed failures on real requests: connection resets, timeouts, consecutive 5xx responses, abnormal latency, or specific protocol errors. This can catch issues that synthetic probes miss.

Suppose /healthz returns 200 because the process is alive, but every request that touches PostgreSQL is timing out because the connection pool is exhausted. Active checks may say healthy. Passive observation says users are suffering.

Good balancers combine both.

Outlier detection removes the "mostly working" backend before it poisons the tail

Modern proxies often implement outlier detection. If one backend starts producing significantly more failures or longer latency than its peers, the proxy ejects it temporarily.

That is valuable because real incidents often begin as partial degradation.

One node in Amsterdam might have:

• a noisy neighbour on the same hypervisor
• packet loss on one TOR uplink
• stale DNS for an internal dependency
• a runaway GC cycle
• a local NVMe problem that makes cache loads stall

The backend is not dead. It is just bad enough to damage user tail latency. Outlier detection gives the proxy a chance to quarantine it for 30 seconds, 60 seconds, or some other ejection window while the system stabilises.

Health signal lag creates its own failure modes

Health is itself a distributed system. If the control plane learns slowly, traffic keeps flowing toward bad nodes for too long. If it reacts too eagerly, brief jitters cause flapping.

Operators therefore tune several thresholds at once:

• how many failures count as suspicious
• how long before a node is ejected
• how often to retry admission
• how many nodes can be ejected at once
• whether to fail open when the healthy pool becomes too small

Fail open versus fail closed is not a theoretical choice. If all health checks fail because the checking system broke or a certificate expired, should the balancer send no traffic at all, or continue using the last known good pool? Different services choose differently. A card authorisation API may prefer a conservative posture. A news site may prefer degraded service over total outage.

Connection draining is how you change the fleet without breaking users

Healthy pools change all the time. Nodes restart, kernels patch, autoscaling adds capacity, old versions drain after deployment, and whole zones may be evacuated for maintenance.

The easy part is adding a node to a pool. The hard part is removing one cleanly.

If the balancer simply deletes a backend immediately, existing users lose live connections. That is acceptable for some short HTTP workloads. It is terrible for:

• WebSockets
• long uploads
• gRPC streams
• SSE connections
• report exports that take minutes

Connection draining solves this by changing the node's state to something like "do not send new work, but allow existing work to finish".

Typical states:

healthy       -> receives new connections or requests
draining      -> keeps old work, rejects new work
unhealthy     -> removed from normal selection
terminated    -> gone

A deployment sequence might be:

1. register new version instances
2. warm local caches and dependency pools
3. pass readiness checks
4. mark old instances draining
5. wait for live requests and connections to fall to zero or timeout threshold
6. terminate old instances

The warm-up step matters more than many teams realise. A new instance that has just started may technically be healthy, yet still slower than the rest because:

• JIT compilation is cold
• caches are empty
• database connections are still being established
• TLS session caches are empty
• model weights or large datasets are still loading

If the balancer immediately gives it a full share of traffic, tail latency jumps. Many systems therefore use slow start or ramp weights gradually.

Example idea in prose: a node joins at weight 10, then 25, then 50, then 100 over a minute. This is not ceremony. It stops a newly started node from looking bad simply because it is new.

Retries can save users or destroy a service, depending on where they fire

Load balancers often retry failed requests automatically. Done carefully, retries hide brief failures from users. Done carelessly, they multiply an outage.

Consider one backend pool where each request has a 700 ms timeout. One dependency starts stalling. The proxy waits 700 ms, retries another backend, waits again, and the client library also retries upstream. Suddenly one user action creates three or four backend attempts.

That is how retry storms begin.

Safe retries depend on idempotency and timing

A GET for a product page is usually safe to retry. A POST that charges a card or creates an order is not safe unless the application has explicit idempotency keys.

The balancer therefore needs policy awareness:

• which methods may be retried
• whether the request body is replayable
• how many attempts are allowed
• which failure types qualify
• how much retry budget exists before the client deadline expires

A good policy is tied to the caller's timeout budget. If the end-user deadline is 1000 ms, two 800 ms backend retries are nonsense. The second attempt arrives too late to matter and only adds pressure.

Hedging is another form of balancing under uncertainty

Some systems use request hedging: after a short delay, a duplicate request is sent to a second backend if the first has not answered yet. The first successful response wins and the loser is cancelled.

This can reduce extreme tail latency on read-heavy idempotent operations. It also increases work. If overused, it becomes self-inflicted load amplification.

The relevant lesson is that a balancer is not only choosing where the first request goes. It may also decide whether a second attempt exists at all. That makes it part scheduler and part damage-control mechanism.

Global load balancing is constrained by routing physics and stale information

Local balancing happens inside one site or one region. Global balancing decides which site the user should hit in the first place.

The usual tools are DNS steering, anycast, or both.

DNS steering is simple, powerful, and slow to converge

A global traffic manager can answer api.example.eu differently depending on geography, latency probes, capacity, or failover state.

Example:

client resolver in Greece -> return Athens region VIP
client resolver in Germany -> return Frankfurt region VIP
if Athens unhealthy -> return Frankfurt instead

The problem is TTL and resolver behaviour. Even if you publish a 30 second TTL, some resolvers cache oddly, some clients reuse connections far longer, and browsers already holding live HTTP/2 sessions do not care that DNS changed until they reconnect.

DNS is therefore good for coarse steering and failover, but it is not an instant steering wheel.

Anycast converges faster for packet ingress, but control is indirect

Anycast advertises the same address from multiple places. BGP decides where the client lands. That gives elegant failover. If one site withdraws the route, traffic shifts elsewhere without the client needing a new DNS answer.

The drawback is control. BGP follows policy and topology, not your product manager's idea of the nearest city. A user in Bucharest might land in Vienna today and Frankfurt tomorrow depending on peering changes.

That is why many systems combine the two:

• DNS chooses the anycast service or product variant
• anycast chooses the physical PoP
• local L4 and L7 balancing choose concrete hosts

Cross-region load balancing has state and data gravity problems

Sending traffic to another region is not free, even when network latency looks acceptable. The backend may depend on:

• a regional database primary
• caches warm only in one city
• legal or residency constraints
• egress cost boundaries
• background jobs consuming local queues

If traffic for a stateful tenant suddenly moves from Paris to Warsaw, the application may still need to read data from Paris. Now the user pays both WAN latency and extra failure surface.

This is why global balancing often cares about affinity at a bigger scale too. The best answer is not always "send to the emptiest region". It may be "keep the tenant near its data unless the region is truly unavailable".

The worst load balancing incidents come from subtle imbalance, not total failure

When a backend goes fully dark, the balancer usually handles it. The nastier incidents involve systems that are alive enough to stay in pool and bad enough to hurt everyone.

Long-lived connections can pin imbalance for hours

WebSockets, gRPC streams, and database proxies create very sticky occupancy. If one set of backends happens to receive a large wave of new long-lived connections during a sporting event or market open, they may stay hotter for hours even if later balancing decisions are fair.

This is one reason operators sometimes rebalance proactively or cap connections per node rather than assuming fair steady state will emerge by itself.

Queue collapse looks balanced until you inspect latency percentiles

Suppose every backend receives about the same request rate, but one third of the requests suddenly become slower because a dependency is lagging. Request counts remain balanced. User experience does not.

The balancer can even make things worse if it keeps treating the slow nodes as healthy peers and feeds them equal traffic. Their queues deepen, latency rises, clients retry, and the extra traffic spills onto the healthy nodes. Soon everything is slow.

This is why observability for load balancing must include:

• per-backend active requests
• request queue depth
• per-backend latency percentiles, not only averages
• ejection counts and health state transitions
• retry counts by reason
• connection counts by protocol version
• success rate split by backend and zone

A mean latency graph over the whole pool is almost useless in this situation. One sick node can disappear inside the average while still dominating the p99 seen by users.

Client-side balancing and proxy-side balancing can fight each other

In microservice stacks, some balancing happens in clients, some in sidecars, some in gateways, and some in DNS or discovery systems. If each layer makes different assumptions, the system oscillates.

Example:

• service discovery hands out all instances equally
• client library sticks to one TCP connection per upstream for efficiency
• proxy retries elsewhere on timeout
• autoscaler adds nodes based on CPU

The new nodes stay underused because existing clients do not reconnect quickly. Old nodes stay hot. Retries increase load. Autoscaler adds even more nodes. The topology changes, but the hot spots remain.

The lesson is mundane and important: adding capacity is not enough if the balancing layer that owns connection placement is too sticky to use it.

The data plane is only half the system. The control plane decides whether it stays sane

When people discuss load balancers, they often focus on request routing algorithms. In production, the control plane is just as important.

The control plane is responsible for:

• distributing backend membership changes
• pushing health state updates
• publishing certificates and listener config
• propagating drain instructions
• updating route rules and weights
• coordinating global steering decisions

If that plane is slow or inconsistent, the data plane behaves strangely.

Examples:

• one proxy knows about 24 backends, another only 18
• one region still thinks a zone is healthy after the other regions removed it
• one fleet has the new canary weight, another does not
• some nodes have the new certificate chain, some do not

These are load balancing problems even though the balancing algorithm itself may be perfect.

This is also where large systems spend serious engineering effort on xDS-style config distribution, membership gossip, health aggregation, and rollout safety. A balancer is a distributed system managing other distributed systems. There is no clean escape from that complexity.

One request path makes the layering easier to see

Abstract lists can make load balancing sound more complicated than it feels on the wire. A concrete trace helps.

Imagine a user in Thessaloniki opening https://shop.example.eu/checkout.

The request path may look like this:

1. DNS returns an anycast service address for the public edge
2. BGP lands the TCP or QUIC flow in the nearest practical PoP
3. an L4 balancer hashes the flow and sends it to frontend fe-07
4. fe-07 terminates TLS and parses the HTTP request
5. the L7 router sees path /checkout and host shop.example.eu
6. it chooses the checkout backend pool, not the catalogue pool
7. the per-pool balancer chooses checkout-12 by least request plus zone preference
8. checkout-12 calls pricing and inventory, each through their own balancing layers
9. the response returns to the frontend and back to the client

Each step is a separate decision point with different failure semantics.

If step 2 changes because BGP reconverges, the client may land in another PoP without any application routing rule changing.

If step 3 changes because one frontend host is withdrawn from the L4 table, only the affected flows should move.

If step 7 changes because checkout-12 starts returning 503s and is ejected by outlier detection, the frontend can keep serving the request path by choosing a different backend from the same pool.

If pricing is slow in step 8, the user may blame "the load balancer" even though the public entry path was perfectly healthy and the real problem sits one tier deeper.

This is why serious incident response almost always asks two questions immediately:

• which balancing layer made the bad decision?
• which layer was correct, but received bad inputs from a dependency or control plane?

Without that split, teams waste hours staring at the wrong dashboards.

A concrete trace also shows why there is no single "load balancer latency" number that explains user experience. The edge proxy may add only 2 ms while the chosen backend queue adds 180 ms. Or the backend may be fast but the wrong region took the traffic because DNS failover lagged. The label is singular. The mechanism is not.

What operators actually tune in practice

The best way to make the topic concrete is to look at the knobs that matter in real deployments. Operators routinely tune things like:

• L4 hash policy inputs
• per-service balancing algorithm
• active health check interval and failure threshold
• passive outlier ejection threshold and duration
• slow-start ramp time for newly admitted instances
• maximum concurrent requests per backend
• retry budget and per-try timeout
• stickiness key and affinity TTL
• circuit breaker limits for connections, pending requests, and retries
• zone-aware routing preferences

A minimal Envoy-style idea might look like this:

lb_policy: LEAST_REQUEST
least_request_lb_config:
  choice_count: 2
outlier_detection:
  consecutive_5xx: 5
  interval: 10s
  base_ejection_time: 30s
slow_start_config:
  slow_start_window: 60s
circuit_breakers:
  thresholds:
    - max_connections: 20000
      max_pending_requests: 5000
      max_retries: 1000

The point is not the exact syntax. The point is that load balancing is mostly policy under operational constraints. The algorithm name alone tells you very little about how the service will behave during a bad afternoon.

Load balancing is a traffic scheduler under failure, not a fairness gadget

That is the cleanest summary I know.

A good load balancer does spread work, but fairness is not the actual goal. The real goals are:

• keep latency low when traffic is uneven
• keep failures small when backends degrade
• shift traffic safely during deploys and outages
• preserve locality when locality matters
• avoid making recovery harder than the original problem

The best designs therefore use different mechanisms at different layers:

• hashing and stable tables for high-speed L4 flow steering
• request-aware algorithms for L7 service selection
• health and outlier systems for partial failure handling
• draining and slow start for safe topology changes
• DNS and anycast for regional or global ingress steering

Once you see load balancing that way, a lot of operational behaviour becomes easier to reason about. A node with equal request share can still be overloaded. A new region can exist and still receive almost no useful traffic. A perfectly healthy process can still be an untrustworthy backend. A retry can be both user-friendly and outage-amplifying, depending on where the time budget comes from.

The box in the architecture diagram was never really a box. It was a scheduler, a policy engine, a health judge, and a traffic-shaping system wearing one label.