Rate Limiting in System Designing

Rate Limiting is a foundational security and traffic-control mechanism used to control the rate of requests a client can make to an API or server within a given timeframe. In system design, a rate limiter acts as a shield, preventing server resources from being exhausted by malicious Distributed Denial of Service (DDoS) attacks, brute-force login attempts, scraping, or poorly written client-side loops ("noisy neighbors"). 1. The Core Implementation Strategies Rate limiting can be applied at different granularities depending on your security architecture: Per-User (Authenticated): Tracks requests based on a unique identifier like a User ID or an API Key (passed via JWT or Authorization headers). This is the most common model for monetization tiers (e.g., Free tier gets 1,000 requests/day, Premium gets 100,000). Per-IP Address (Unauthenticated): Tracks requests based on the client's public IP address. Useful for protecting public, unauthenticated endpoints like /login or /register . Downside: Can accidentally throttle an entire office building or school sharing a single NAT gateway IP. Per-Endpoint: Rates are tuned to specific resource costs. For example, a lightweight GET /search might allow 100 requests per minute, while a computationally expensive POST /generate-pdf might be limited to 2 requests per minute. 2. Core Rate Limiting Algorithms Choosing the right algorithm alters how your system handles temporary traffic bursts. A. Token Bucket How it works: A bucket has a maximum capacity of $N$ tokens. It is continuously refilled with tokens at a constant rate over time. Every incoming request consumes exactly one token. If the bucket is empty, the request is dropped. Characteristics: Highly efficient memory-wise. It natively allows for bursts of traffic (if the bucket is completely full, a client can send $N$ requests simultaneously without issue), making it the industry standard for APIs. B. Leaky Bucket How it works: Imagine a bucket with a small hole at the bottom. Requests enter the bucket at arbitrary rates, but they leak out (are processed by your server) at a strict, continuous, constant rate. If the bucket fills up with requests faster than it leaks, the excess requests overflow and are dropped immediately. Characteristics: It smooths out traffic into a perfectly steady output rate , eliminating bursts entirely. Excellent for scenarios where backend systems require predictable, continuous load leveling. C. Sliding Window Log / Counter How it works: Instead of rigid 1-minute blocks (which suffer from boundary attacks where a user doubles their traffic right at the edge of a minute change), the sliding window tracks the exact timestamp of every request relative to the current exact millisecond. Characteristics: The most accurate way to enforce strict time-window limits. However, it requires storing a high volume of request timestamps in memory (usually in a Redis Sorted Set ), making it highly memory-intensive for high-traffic platforms. 3. The HTTP Handshake: 429 Too Many Requests When a client breaches their allocated rate limit, the rate limiter intercepts the request before it hits the application server and returns a standard 429 Too Many Requests HTTP status code. To build a professional API contract, the server should pass metadata back to the client inside the response headers to help them back off gracefully: HTTP HTTP/1.1 429 Too Many Requests Content-Type: application/json Retry-After: 30 X-RateLimit-Limit: 100 X-RateLimit-Remaining: 0 X-RateLimit-Reset: 1717430220 { "error": "Rate limit exceeded. Please try again later." } X-RateLimit-Limit : The maximum number of allowed requests in the current period. X-RateLimit-Remaining : The number of remaining requests allowed within the current window (will be 0 on a 429). X-RateLimit-Reset : The Unix epoch timestamp indicating exactly when the current rate limit window resets. Retry-After : The number of seconds the client must wait before trying again (an alternative or supplement to the reset timestamp). 4. Structural Architecture Placement In production environments, you do not write rate-limiting code directly inside your core application logic (like your Node.js or Go business code). Doing so wastes CPU cycles on requests you intend to drop anyway. Instead, rate limiting is placed at the Edge Layer : API Gateway / Reverse Proxy (NGINX, HAProxy, AWS API Gateway): Intercepts traffic at the perimeter. Centralized Memory Store (Redis): Because your API Gateways might scale horizontally, they all query a central, ultra-fast Redis cluster using atomic scripts ( INCR or Lua scripts) to track the current token counts for each user ID across the whole network instantly.