What is Load Balancer in System Architecture

Load Balancing Basics When a stateless service scales out horizontally into a pool of multiple independent server instances, you need a mechanism to distribute incoming user traffic across those nodes. If all web requests hit only the first server in your cluster, that machine will quickly run out of CPU and RAM, while the other servers sit idle. A Load Balancer (LB) is a specialized network intermediary that distributes incoming application traffic across a healthy pool of backend servers, preventing any single machine from overloading and ensuring high availability. 1. The Gateway: What is a Load Balancer? A load balancer acts as a single point of entry for your infrastructure. It exposes a single public IP address to the internet, catches all incoming customer requests, and balances them out across your private backend server pool. LOAD BALANCER TRAFFIC ROUTING ┌───────────────┐ ┌──────────────────┐ │ │ ─────► │ Backend Server 1 │ PUBLIC CLIENTS ───────────────► │ Load Balancer │ ├──────────────────┤ (Traffic Stream) │ (Single IP) │ ─────► │ Backend Server 2 │ │ │ ├──────────────────┤ └───────────────┘ ─────► │ Backend Server 3 │ └──────────────────┘ Core Architectural Mandates High Availability & Failover: It isolates the client from individual server crashes. If a backend node fails, the load balancer stops routing traffic to it automatically. Elastic Scaling Integration: As auto-scaling groups spin up new server instances during traffic spikes, they register directly with the load balancer to start taking on work seamlessly. Security Shielding: It conceals the internal private IP addresses and exact counts of your backend infrastructure from the open web. To process and route traffic efficiently, load balancers operate at different layers of the standard OSI (Open Systems Interconnection) Model . The two primary types used in enterprise systems are Layer 4 (L4) and Layer 7 (L7) load balancers. 2. Transport-Layer Routing: Layer 4 Load Balancer A Layer 4 Load Balancer operates purely at the Transport Layer of the network stack. It routes data packets blindly based exclusively on low-level network information, such as the source/destination IP Address and the TCP/UDP Port number . Production Mechanics An L4 load balancer does not look at, parse, or understand the contents of the application data passing through it. It treats every request as a stream of raw network packets. When a packet arrives, the load balancer quickly modifies the packet's destination IP header to match one of its backend servers using fast NAT (Network Address Translation) techniques, and forwards the raw stream down the line. Core Trade-offs Pros (Ultra-High Speed): Because it skips parsing heavy application data formats like HTTP headers, cookies, or JSON payloads, it requires minimal CPU processing power. This allows L4 load balancers to route millions of concurrent connections at wire-speed with exceptionally low latency. Cons (No Content Awareness): It cannot make smart routing choices. For example, it cannot route traffic based on a user's login status, a specific URL path string, or a device type, because it has no way of reading that data. 3. Application-Layer Routing: Layer 7 Load Balancer A Layer 7 Load Balancer operates at the top Application Layer of the network stack. It possesses full visibility into the actual data content moving inside the network traffic stream. Production Mechanics To read application data, an L7 load balancer must perform TCP Connection Termination . It terminates the client's incoming TCP connection directly on itself, decrypts the SSL/TLS layer, parses the underlying HTTP/HTTPS protocol headers, and reads the full request payload. Once it inspects the request details, it opens a brand-new, independent internal TCP connection to forward the payload to the most appropriate specialized backend microservice pool. Advanced Smart Routing Capabilities Path-Based Routing: You can route requests based entirely on the URL path structure. For example, requests to example.com/api/v1/billing can be routed to a heavy, memory-optimized billing server cluster, while requests to example.com/static/* bypass application nodes completely and route straight to a localized asset cache. Cookie-Based Sticky Sessions: Reads session identifier cookies to ensure an active user is consistently routed to the exact same backend server instance for the duration of their session. Header-Based Inspections: Routes traffic based on the user's User-Agent string (separating mobile traffic from desktop traffic) or checks authentication headers directly at the entrance boundary. 4. System Gatekeeping: Health Checks A load balancer is only effective if it routes traffic to fully operational servers. If a backend application server suffers an unhandled database connection crash, it might still be powered on and listening on its port, but it will return 500 Server Errors to every incoming request. To prevent routing traffic to broken instances, load balancers use automated Health Checks . AUTOMATED HEALTH CHECK PROBE LOOP ┌───────────────┐ 1. HTTP GET /healthz Probe ┌──────────────────┐ │ │ ──────────────────────────────────────────► │ Backend Server │ │ Load Balancer │ │ (Checks DB, │ │ │ ◄────────────────────────────────────────── │ Memory, RAM) │ └───────────────┘ 2. HTTP 200 OK Status Return └──────────────────┘ (Server Cleared as HEALTHY) How the Health Check Engine Executes The load balancer runs an automated, continuous background loop using explicit testing parameters configured by an engineer: The Probe Target: The load balancer sends periodic ping requests (such as an HTTP GET /healthz or a raw TCP connection probe) to every registered backend instance. The Internal Status Logic: The backend application shouldn't just return a dummy text file when hit on its health endpoint. It should actively check its own internal health—verifying that its primary database connections are active, its local disk space isn't full, and its memory allocation is stable. If all internal checks pass, it returns an explicit 200 OK status code. The Eviction Threshold: If a server fails a health probe (e. g., times out, drops connection, or returns a 500 error) for a configured number of consecutive attempts (e. g., 3 failed checks), the load balancer flags the node as Unhealthy . Traffic Rerouting: The load balancer instantly evicts the unhealthy node from its active routing pool, protecting users from landing on a broken server. The health check loop continues pinging the broken instance in the background; as soon as the node passes enough consecutive checks to prove it is stable again, the load balancer gracefully brings it back into the rotation. Load Balancing Architecture Reference Matrix Feature Layer Vector Layer 4 (L4) Transport Load Balancer Layer 7 (L7) Application Load Balancer OSI Model Placement Layer 4 (TCP / UDP Transport Streams). Layer 7 (HTTP / HTTPS / gRPC Protocols). Data Visibility Horizon Blind. Reads only IP coordinates and port numbers. Deep. Inspects URLs, cookies, HTTP headers, and message payloads. Processing Latency Ultra-Low. Fast packet switching with minimal CPU utilization overhead. Moderate. Requires terminating TCP links and parsing text strings. Cryptographic SSL/TLS Duties Bypasses decryption; forwards encrypted packets raw. Active. Handles SSL/TLS termination and certificates right at the network boundary. Primary Production Role High-volume raw entry routers, gaming pipelines, or database cluster routing. Microservice architectures, path-routing configurations, API Gateways, and SaaS apps.