Rate Limiting

Why Rate Limit

A bot that hits POST /api/orders 500 times a second, or a buggy app stuck in a loop, can take down order-service for everyone. The gateway can limit each client (by user ID, API key or IP). Extra requests get 429 Too Many Requests without touching the service.

Spring Cloud Gateway uses the token bucket algorithm: each key has a bucket holding up to burstCapacity tokens, refilled at replenishRate tokens per second. A request takes one token; an empty bucket means 429.

Token Bucket in Plain Java

Example

A bucket with capacity 5 and refill 2 tokens/second, with a simulated clock in milliseconds:

public class Main {
    public static void main(String[] args) {
        TokenBucket bucket = new TokenBucket(5, 2.0); // burst 5, refill 2/s
        long[] requestTimesMs = {0, 0, 0, 0, 0, 0, 0, 1000, 1000, 1000, 2500, 2500, 2500, 2500};

        int n = 1;
        for (long t : requestTimesMs) {
            boolean allowed = bucket.tryConsume(t);
            System.out.printf("t=%4dms  request %2d -> %-22s (tokens left: %d)%n",
                    t, n++, allowed ? "200 OK" : "429 Too Many Requests", (int) bucket.tokens());
        }
    }
}

class TokenBucket {
    private final int capacity;
    private final double refillPerSecond;
    private double tokens;
    private long lastRefillMs = 0;

    TokenBucket(int capacity, double refillPerSecond) {
        this.capacity = capacity;
        this.refillPerSecond = refillPerSecond;
        this.tokens = capacity;                     // starts full
    }

    boolean tryConsume(long nowMs) {
        double elapsedSeconds = (nowMs - lastRefillMs) / 1000.0;
        tokens = Math.min(capacity, tokens + elapsedSeconds * refillPerSecond);
        lastRefillMs = nowMs;
        if (tokens >= 1) {
            tokens -= 1;
            return true;
        }
        return false;
    }

    double tokens() { return tokens; }
}

Output

t=   0ms  request  1 -> 200 OK                 (tokens left: 4)
t=   0ms  request  2 -> 200 OK                 (tokens left: 3)
t=   0ms  request  3 -> 200 OK                 (tokens left: 2)
t=   0ms  request  4 -> 200 OK                 (tokens left: 1)
t=   0ms  request  5 -> 200 OK                 (tokens left: 0)
t=   0ms  request  6 -> 429 Too Many Requests  (tokens left: 0)
t=   0ms  request  7 -> 429 Too Many Requests  (tokens left: 0)
t=1000ms  request  8 -> 200 OK                 (tokens left: 1)
t=1000ms  request  9 -> 200 OK                 (tokens left: 0)
t=1000ms  request 10 -> 429 Too Many Requests  (tokens left: 0)
t=2500ms  request 11 -> 200 OK                 (tokens left: 2)
t=2500ms  request 12 -> 200 OK                 (tokens left: 1)
t=2500ms  request 13 -> 200 OK                 (tokens left: 0)
t=2500ms  request 14 -> 429 Too Many Requests  (tokens left: 0)

A burst of 5 passes at once, then the client can only do 2 requests per second on average. That is exactly what burstCapacity: 5 and replenishRate: 2 mean.

RequestRateLimiter with Redis

Every gateway instance must share the same buckets, so Spring stores them in Redis. Add spring-boot-starter-data-redis-reactive, then:

spring:
  data:
    redis:
      host: localhost
      port: 6379
  cloud:
    gateway:
      routes:
        - id: order-service
          uri: lb://order-service
          predicates:
            - Path=/api/orders/**
          filters:
            - name: RequestRateLimiter
              args:
                redis-rate-limiter.replenishRate: 2
                redis-rate-limiter.burstCapacity: 5
                redis-rate-limiter.requestedTokens: 1
                key-resolver: "#{@userKeyResolver}"

@Configuration
class RateLimitConfig {

    /** One bucket per logged-in user; fall back to the client IP. */
    @Bean
    KeyResolver userKeyResolver() {
        return exchange -> exchange.getPrincipal()
                .map(Principal::getName)
                .switchIfEmpty(Mono.fromSupplier(() ->
                        exchange.getRequest().getRemoteAddress().getAddress().getHostAddress()));
    }
}

Example

The 6th request in the same second is rejected:

for i in 1 2 3 4 5 6; do curl -s -o /dev/null -w "%{http_code} " -X POST localhost:8080/api/orders -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d @order.json; done

Example output

201 201 201 201 201 429

HTTP/1.1 429 Too Many Requests
X-RateLimit-Remaining: 0
X-RateLimit-Burst-Capacity: 5
X-RateLimit-Replenish-Rate: 2
X-RateLimit-Requested-Tokens: 1

Exercise

Exercise: Change the TokenBucket example so expensive requests (e.g. POST /api/orders) cost 3 tokens and cheap ones cost 1. Add a tryConsume(long nowMs, int cost) method.

Show answer

boolean tryConsume(long nowMs, int cost) {
    double elapsedSeconds = (nowMs - lastRefillMs) / 1000.0;
    tokens = Math.min(capacity, tokens + elapsedSeconds * refillPerSecond);
    lastRefillMs = nowMs;
    if (tokens >= cost) {
        tokens -= cost;
        return true;
    }
    return false;
}
// In the gateway this is redis-rate-limiter.requestedTokens: 3 on the order route.