How a Load Balancer Works? A Simple Guide for Software Engineers

Preparing for System Design Interviews? Join ByteByteGo now for a more structured preparation. They are also offering a huge 50% discount now on their lifetime plan.

Hello guys, Load balancer and API Gateway is one of the most essential System Design concepts and also a popular topic on System Design Interview. A good knowledge of these concepts not just help you do well on interview but also on solving production issues and designing an architecture which can withstand test of time in production.

Imagine your application suddenly starts receiving 100,000 requests per second.

One server probably won’t be able to handle that workload reliably. Even if you buy a much larger server, you eventually hit limits around CPU, memory, network bandwidth, or connection capacity.

So instead of making one server bigger, you can add more servers and distribute traffic between them.

This is where a Load Balancer comes in.

At a high level, the architecture looks like this:

                         ┌──────────────┐
                         │    Users     │
                         └──────┬───────┘
                                │
                         100,000 req/sec
                                │
                                ▼
                       ┌─────────────────┐
                       │  Load Balancer  │
                       └────────┬────────┘
                                │
                ┌───────────────┼───────────────┐
                │               │               │
                ▼               ▼               ▼
          ┌──────────┐    ┌──────────┐    ┌──────────┐
          │ Server 1 │    │ Server 2 │    │ Server 3 │
          └──────────┘    └──────────┘    └──────────┘
                                │
                                ▼
                         ┌──────────────┐
                         │   Server 4   │
                         └──────────────┘

The load balancer sits between clients and your application servers.

Instead of users connecting directly to a particular server, they connect to the load balancer.

The load balancer then decides:

Which server should handle this request?

That sounds simple, but there is quite a lot happening behind the scenes.

Let’s break it down.

A load balancer is a component that distributes incoming network traffic across multiple servers.

Without a load balancer, you might have:

 User
  │
  ▼
Server

If millions of users arrive, that single server becomes a bottleneck.

With a load balancer, architecture look like below: