Load Balancing: How Traffic Gets Distributed Across Servers

⏱️ Lectura: 12 min

A server going down shouldn’t take your application down with it, and deciding in milliseconds which of the hundreds of available servers should receive each request is the job of a single piece of infrastructure: the load balancer.

📑 En este artículo

1. TL;DR
2. What load balancing is and why it matters
3. How load balancing works under the hood
   1. Layer 4 vs layer 7
   2. Distribution algorithms
   3. Health checks
4. Practical examples: configuring a real load balancer
5. How to get started step by step
6. Real-world use cases
7. Common mistakes and best practices
8. Comparison: balancing algorithms
9. Going deeper: advanced balancing
10. Frequently Asked Questions
    1. What’s the difference between load balancing and a reverse proxy?
    2. Do I need load balancing if I have few users?
    3. Is round robin always the best option?
    4. What happens if the load balancer itself goes down?
    5. Nginx or HAProxy?
    6. Does load balancing replace a CDN?
11. References

From the site you visit every day to the APIs behind a banking app, almost all web traffic passes through a load balancer before reaching the server that actually generates the response. Understanding how that balancing works means understanding how the internet scales.

TL;DR

• You’ll understand how a load balancer decides which server should receive each request in milliseconds.
• You’ll be able to configure a real load balancer with Nginx (upstream) and HAProxy (backend) step by step.
• You’ll learn to distinguish round robin, least connections, IP hash, and consistent hashing, and when to use each one.
• You’ll know how to implement health checks to automatically pull down servers out of rotation.
• You’ll differentiate layer 4 (TCP) balancing from layer 7 (HTTP) balancing and why it matters.
• You’ll identify the common mistakes that break user sessions when balancing traffic.
• You’ll be able to verify with concrete commands whether your load balancer is routing as expected.

What load balancing is and why it matters

Load balancing is the technique of distributing incoming requests across multiple servers so that none of them gets overloaded and, if one fails, the others keep responding. Instead of exposing a single server to the internet, you expose an address or domain that points to the load balancer, and it decides, request by request, where to forward the traffic.

The underlying reason is twofold: capacity and availability. A single server has a ceiling on CPU, memory, and bandwidth; adding replicas behind a load balancer lets you scale horizontally without touching the application code. And if a server goes down, whether from a deployment, a crash, or a network issue, the load balancer stops sending it traffic and nobody on the client side notices the difference.

Practically all modern infrastructure applies load balancing at more than one layer at once: a cloud load balancer distributes traffic across availability zones, and within each zone an Nginx or HAProxy instance distributes that traffic across the application’s instances. In microservices architectures, a service mesh like Envoy adds a third balancing layer between the internal services themselves.

How load balancing works under the hood

Layer 4 vs layer 7

A load balancer can operate at layer 4 (transport, TCP/UDP) or layer 7 (application, HTTP). A layer 4 balancer only looks at the source and destination IP and port: it’s fast because it doesn’t need to understand the packet’s content, but it can’t make decisions based on the request’s URL, headers, or cookies. A layer 7 balancer, on the other hand, terminates the HTTP connection, reads the full request, and can route /api to one group of servers and /static to another, or always send an authenticated user to the same backend.

Nginx and HAProxy can operate in both modes depending on the configuration; cloud load balancers like a Network Load Balancer work at layer 4, while an Application Load Balancer works at layer 7.

Distribution algorithms

The heart of the load balancer is the algorithm that decides, for each new request, which server receives it. The most common ones are round robin (taking turns), least connections (fewest active connections), IP hash (the same IP always goes to the same server), and consistent hashing, a variant designed to minimize remapping when a server is added or removed, the same technique behind systems like Cassandra.

The following diagram shows where the load balancer sits between the client and the backend servers:

flowchart TD
 A["Client"] --> B["Load Balancer"]
 B --> C["Server 1"]
 B --> D["Server 2"]
 B --> E["Server 3"]
 subgraph Backend
 C
 D
 E
 end

Health checks

No algorithm is any good if the load balancer doesn’t know which servers are alive. Health checks are periodic requests, a GET to /health for example, that the load balancer sends to each backend server; if a server fails to respond or responds with an error for several checks in a row, the load balancer automatically pulls it out of rotation and stops sending it traffic until it starts responding correctly again.

Practical examples: configuring a real load balancer

To bring the theory down to earth, here’s what load balancing looks like configured with the two most widely used open source tools for this. This Nginx block defines a group of three backend servers with different weights and balancing by fewest active connections:

upstream backend_app {
 least_conn;
 server 10.0.0.11:8080 weight=3;
 server 10.0.0.12:8080 weight=2;
 server 10.0.0.13:8080 backup;
}

server {
 listen 80;
 location / {
 proxy_pass http://backend_app;
 proxy_next_upstream error timeout http_502;
 }
}

least_conn tells Nginx to prioritize the server with the fewest open connections instead of distributing in blind round-robin fashion; weight=3 makes that server receive proportionally more traffic than one with weight=2, and backup marks a server that only receives traffic if the other two fail. proxy_next_upstream defines in which cases Nginx retries the request against another server in the group instead of returning the error to the client.

HAProxy expresses the same thing with its own configuration language, adding an explicit HTTP health check:

frontend http_front
 bind *:80
 default_backend app_servers

backend app_servers
 balance roundrobin
 option httpchk GET /health
 server web1 10.0.0.11:8080 check weight 3
 server web2 10.0.0.12:8080 check weight 2
 server web3 10.0.0.13:8080 check backup

option httpchk GET /health activates the check against that route on each server; the word check on each server line enables that specific check, and if the server stops responding with a 200, HAProxy pulls it out of rotation without manual intervention.

How to get started step by step

To try this on your own machine, all you need is Docker and a couple of test servers.

1. Install HAProxy: apt install haproxy on Debian or Ubuntu, or brew install haproxy on macOS.
2. Spin up three test servers: python3 -m http.server 8080 repeated on three different ports, or three separate containers.
3. Paste the configuration above into /etc/haproxy/haproxy.cfg and restart the service with systemctl restart haproxy.
4. Confirm it distributes traffic with several requests in a row: for i in 1 2 3 4 5 6; do curl -s http://localhost/ | head -1; done.
5. Turn off one of the test servers and repeat the curl loop to confirm that the load balancer stops sending it traffic within a few seconds.
6. Enable the stats dashboard by adding this block to the configuration:

listen stats
 bind *:8404
 stats enable
 stats uri /stats

and visiting http://localhost:8404/stats in the browser, where each server shows up in green (UP) or red (DOWN) in real time.

Real-world use cases

Load balancing shows up in layers that are rarely visible from the outside. A typical cloud web application uses an application load balancer to distribute HTTP traffic across instances in different availability zones, so that if an entire zone fails, the rest keep serving traffic.

In database architectures, a read balancer distributes SELECT queries across several replicas while writes always go to the primary node; this is different from API balancing, because here the criterion isn’t just capacity but also how up to date each replica is.

At the global level, DNS also performs a primitive form of balancing: the same domain can resolve to several different IPs (DNS round robin), and CDN providers use Anycast so the same IP responds from the datacenter geographically closest to the user, without any single centralized load balancer.

Inside a container cluster, Kubernetes exposes a Service that balances internally across the pods backing an application, and a service mesh like Envoy adds an extra balancing layer between the internal services themselves, with the advantage that each instance knows the latency and status of its neighbors in real time.

Common mistakes and best practices

• Ignoring stateful sessions: if your app stores sessions in the server’s local memory and the load balancer doesn’t use cookie or IP affinity, the user loses their session every time a request lands on a different server.
• Shallow health checks: a health endpoint that only confirms the process responds, but not that it can connect to the database, leaves servers alive but useless inside the rotation.
• A single load balancer as a single point of failure: if the load balancer doesn’t have its own redundancy, active-passive with keepalived or a managed cloud load balancer, all the fault tolerance of the backend servers is worthless.
• Poorly calibrated timeouts: a timeout that’s too long makes the load balancer keep waiting on a slow server instead of retrying against another one, dragging the slowness down onto every user.
• Ignoring weights on heterogeneous servers: splitting traffic evenly between a server with double the CPU and a smaller one wastes capacity; that’s where weight makes the difference.

⚠️ Watch out: a health check that only returns 200 without verifying real dependencies, like database, cache, or disk, can keep a broken server inside the rotation for hours.

Comparison: balancing algorithms

Algorithm	When to use it	Advantage	Limitation
Round Robin	Servers with similar capacity	Simple, even distribution	Ignores each server’s real load
Least Connections	Requests with variable duration	Avoids overloading the slowest server	Requires keeping state of active connections
IP Hash	Apps with local in-memory sessions	The same client always goes to the same server	Uneven distribution if few clients generate a lot of traffic
Weighted	Servers with different capacity	Makes use of heterogeneous hardware	Weights have to be adjusted manually
Consistent Hashing	Clusters that scale frequently	Minimal remapping when adding or removing nodes	More complex to implement from scratch

Going deeper: advanced balancing

Consistent hashing solves a specific problem with IP hash: when you add or remove a server, a simple hash remaps almost all the keys to different servers, which breaks caches and sessions in cascade. Consistent hashing distributes servers and keys on the same ring, so adding or removing a node only remaps a small fraction of the traffic instead of all of it.

Google published its own algorithm for this problem, Maglev, designed for software load balancers that handle millions of connections without maintaining a per-connection state table: each packet is routed consistently using only a hash of its headers, so one load balancer can go down and another take its place without cutting existing connections.

Here’s what the sequence of a health check followed by a real request looks like, step by step:

sequenceDiagram
 participant C as Client
 participant LB as Load Balancer
 participant S1 as Server A
 participant S2 as Server B
 LB->>S1: health check
 S1-->>LB: OK 200
 LB->>S2: health check
 S2-->>LB: timeout
 Note over LB,S2: Server B marked as down
 C->>LB: requests page
 LB->>S1: forwards request
 S1-->>LB: returns data
 LB-->>C: returns data

The health check runs in the background, independent of any user request; when a server fails that check, the load balancer removes it from the list of candidates before any real client reaches it.

💡 Tip: to confirm live which algorithm your Nginx is using, check the directive inside the upstream block with nginx -T | grep -A5 upstream; if you don’t see least_conn or ip_hash, it’s using round robin by default.

📖 Summary on Telegram: View summary

Your next step: spin up three containers with a simple HTTP server, paste this article’s HAProxy configuration into /etc/haproxy/haproxy.cfg, and watch the /stats dashboard while you turn off one of the three by hand.

Frequently Asked Questions

What’s the difference between load balancing and a reverse proxy?

A reverse proxy forwards requests to a single backend on the client’s behalf; a load balancer is a reverse proxy that additionally chooses between several possible backends using a distribution algorithm. Every HTTP load balancer is, technically, a reverse proxy with extra logic.

Do I need load balancing if I have few users?

Not for capacity, but it can still be worth it for availability: even with little traffic, a load balancer with two identical servers prevents a deployment or a crash from taking down the entire site while the other server keeps responding.

Is round robin always the best option?

No. Round robin assumes that all requests cost the same and that all servers have the same capacity; in practice that’s rarely true, and least connections or weighted tend to give better results.

What happens if the load balancer itself goes down?

It becomes a single point of failure. That’s why in production you run at least two load balancers in active-passive mode with a shared virtual IP (for example with keepalived), or use a load balancer managed by the cloud provider that already handles that redundancy.

Nginx or HAProxy?

Nginx is more common when the load balancer also serves as a web server or proxy for static content; HAProxy is designed from the ground up exclusively for balancing and tends to give finer control over health checks and connection queues.

Does load balancing replace a CDN?

No, they solve different problems. A CDN caches content close to the user to reduce geographic latency; a load balancer distributes requests across your own origin servers. Many architectures use both at once.

References

• Wikipedia: Load balancing (computing): general definition and classic algorithms.
• Official Nginx documentation: supported balancing methods and configuration directives.
• Official HAProxy site: project and load balancer documentation.
• Kubernetes: Service: how Kubernetes balances traffic across pods.

📱 Enjoying this content? Follow @programacion on Telegram for daily tech content in Spanish: quick summaries, fresh content every day.

Imagen destacada: Foto de Winston Chen en Unsplash