Load Balancing in System Designing

A Load Balancer acts as the single point of entry and the traffic cop for your application. When a system scales horizontally, you deploy multiple application servers to handle the load. The load balancer sits in front of these servers, accepting all incoming client requests and routing them across the server pool. This ensures no single server becomes overwhelmed, maximizing throughput and reducing latency. 1. Common Load Balancing Algorithms The choice of algorithm determines the strategy the load balancer uses to distribute incoming traffic: Round Robin: Requests are distributed sequentially across the list of available servers (e.g., Server 1, then Server 2, then Server 3, then back to Server 1). Best for: Systems where all backend servers have identical hardware capabilities and tasks take roughly the same amount of time to process. Least Connections: Directs traffic to whichever server currently has the fewest active, open client connections. Best for: Applications where requests vary wildly in processing time (e.g., one user is downloading a large file while another is loading a lightweight profile page). IP Hash: Computes a hash key using the client's IP address and maps it to a specific server. Best for: Applications requiring Session Persistence (Sticky Sessions) . This ensures that a specific user always lands on the exact same server, which is useful if the backend relies on local server memory to store user sessions (though stateless architecture is still preferred for high scale). 2. Layer 4 vs. Layer 7 Load Balancing Load balancers operate at different layers of the OSI model, changing how much insight they have into the traffic passing through them. Feature Layer 4 (Transport Layer) Layer 7 (Application Layer) Data Visibility Inspects only TCP/UDP protocol headers, source IP, and ports. It cannot see the request content. Full visibility into the HTTP/HTTPS payload, including URLs, Cookies, and Headers. Routing Decisions Fast, raw network-level packet routing. Smart, application-aware routing (e.g., sending /api requests to a backend pool and /images to an asset server). Encryption Cannot view encrypted data. Passes the SSL tunnel through to the servers. Handles TLS/SSL Termination (decrypts the request at the balancer before passing plaintext to internal servers). Resource Usage Extremely lightweight; requires very little CPU. More resource-intensive because it must parse and rebuild full application-layer messages. 3. Industry Standard Tools NGINX: A wildly popular open-source reverse proxy that doubles as an efficient load balancer. It excels at Layer 7 routing and handling massive volumes of concurrent connections with low memory usage. HAProxy: High Availability Proxy. A dedicated, ultra-fast load balancer known for its incredible raw performance, stability, and deep metrics configuration at both Layer 4 and Layer 7. AWS ALB (Application Load Balancer): A fully managed, auto-scaling Layer 7 load balancer provided by Amazon Web Services. It integrates natively with cloud-native containers and EC2 instances, abstracting away underlying server maintenance. 4. System Design Impact: Avoiding the Single Point of Failure (SPOF) While a load balancer protects your application servers from crashing, the load balancer itself becomes a Single Point of Failure . If your lone load balancer machine goes offline, your entire system goes dark. To prevent this in production systems, you deploy load balancers in a High Availability (HA) Cluster : An Active load balancer handles 100% of the live traffic. A Passive (Standby) load balancer sits idly next to it, constantly monitoring the active balancer via a background heartbeat connection. If the active balancer fails, the passive balancer instantly detects the crash, assumes the shared Virtual IP address, and seamlessly takes over the traffic stream with zero downtime to the end users.