Day9 in System Designing

In system design, optimizing infrastructure requires a deep understanding of performance metrics. Two metrics form the absolute foundation of system evaluation: Latency and Throughput . While they are closely related, they measure entirely different aspects of system capacity. 1. Latency: The Clock Metric Latency is the time it takes for a single data packet or request to travel from the source to the destination and return a response. It is a measurement of delay and is typically measured in milliseconds (ms). The Contributors to Latency: Propagation Delay: The time it takes for a signal to travel through physical space (limited by the speed of light in fiber-optic cables). Transmission Delay: The time required to push the data packets onto the physical network medium (determined by the size of the packet and the connection medium). Processing Delay: The time your backend server takes to compute business logic, run V8 JavaScript execution, or parse JSON. Queuing Delay: The time a request spends sitting in a load balancer or a thread pool queue waiting for a worker to become available. 2. Throughput: The Volume Metric Throughput is the number of data units or requests a system can successfully process within a specific time window. In web applications, it is commonly measured in RPS (Requests Per Second) or QPS (Queries Per Second) . While latency tracks how fast a single user's request completes, throughput tracks how much total work the entire server cluster can handle simultaneously. 3. Bandwidth  ≠    Throughput A common point of confusion in network design is conflating bandwidth with throughput. Bandwidth: The theoretical maximum capacity of a network link. It is the width of the pipe. Throughput: The actual amount of data successfully flowing through that pipe under real-world conditions. The Highway Analogy: > Think of a highway network. Bandwidth is the number of lanes available. Latency is the time it takes for a single car to drive from City A to City B. Throughput is the actual number of cars passing a specific checkpoint per hour. If there is an accident (network congestion) or a slow toll booth (slow database queries), the cars slow down (high latency), and fewer cars pass per hour (low throughput), even though the highway still has 4 open lanes (high bandwidth). 4. The Core Goal: Low Latency + High Throughput When designing an architecture, your ultimate objective is to maintain a system that exhibits low latency (snappy user experience) and high throughput (highly scalable under heavy traffic load). However, these metrics do not always scale linearly. As traffic grows, they interact in predictable phases: Phase A: The Sweet Spot When user load is low to moderate, throughput increases linearly with traffic, while latency remains flat and low. The system has plenty of available CPU cores, thread pools, and open database connections. Phase B: The Saturation Point As traffic approaches your infrastructure limits, throughput plateaus . The system is handling the maximum number of requests it physically can per second. At this exact moment, latency begins to spike drastically . Incoming requests are forced to wait in OS queues, reverse proxy buffers, or libuv thread pools. 5. System Design Strategies to Fix Both When a system breaks under load, you use different architectural levers depending on which metric is failing: To Fix High Latency: Implement Caching (Redis) to avoid hitting slow disk-based databases. Use a Content Delivery Network (CDN) to store static assets physically closer to users, cutting down network propagation delay. Optimize database queries and introduce indexing. To Fix Low Throughput: Scale horizontally by adding more server instances behind a Load Balancer . Introduce Asynchronous Task Queues (BullMQ, RabbitMQ) to offload non-blocking work from the main request thread, freeing up the application to accept the next inbound connection instantly.