Client-side Load Balancing

Spring Cloud LoadBalancer

Eureka gives you a list. Spring Cloud LoadBalancer picks one instance per request. It comes with the Eureka client starter. Mark a RestClient.Builder with @LoadBalanced and use the service name as the host:

@Configuration
class HttpClientConfig {

    @Bean
    @LoadBalanced                      // "product-service" -> a real host:port
    RestClient.Builder loadBalancedRestClientBuilder() {
        return RestClient.builder();
    }
}

@Service
class ProductClient {

    private final RestClient restClient;

    ProductClient(RestClient.Builder builder) {
        this.restClient = builder.baseUrl("http://product-service").build();
    }

    ProductDto getProduct(long id) {
        return restClient.get()
                .uri("/api/products/{id}", id)
                .retrieve()
                .body(ProductDto.class);
    }
}

Note: @LoadBalanced works the same way on RestTemplate and WebClient.Builder. With OpenFeign (later module) load balancing is automatic.

Round Robin, Step by Step

The default algorithm is round robin: instance 1, 2, 3, 1, 2, 3... Here is the same logic in plain Java with an in-memory registry. When an instance leaves the registry, the balancer simply stops choosing it:

Example

import java.util.*;
import java.util.concurrent.atomic.AtomicInteger;

public class Main {
    public static void main(String[] args) {
        ServiceRegistry registry = new ServiceRegistry();
        registry.register(new Instance("product-service", "10.0.0.11", 8081));
        registry.register(new Instance("product-service", "10.0.0.12", 8081));
        registry.register(new Instance("product-service", "10.0.0.13", 8081));

        RoundRobinLoadBalancer lb = new RoundRobinLoadBalancer(registry);
        for (int i = 1; i <= 6; i++) {
            System.out.println("Request " + i + " -> " + lb.choose("product-service").url());
        }

        System.out.println("--- 10.0.0.12 stops sending heartbeats and is removed ---");
        registry.deregister("product-service", "10.0.0.12");
        for (int i = 7; i <= 10; i++) {
            System.out.println("Request " + i + " -> " + lb.choose("product-service").url());
        }
    }
}

record Instance(String serviceId, String host, int port) {
    String url() { return "http://" + host + ":" + port; }
}

class ServiceRegistry {
    private final Map<String, List<Instance>> apps = new TreeMap<>();

    void register(Instance i) {
        apps.computeIfAbsent(i.serviceId(), k -> new ArrayList<>()).add(i);
    }
    void deregister(String serviceId, String host) {
        apps.getOrDefault(serviceId, List.of()).removeIf(i -> i.host().equals(host));
    }
    List<Instance> instances(String serviceId) {
        return apps.getOrDefault(serviceId, List.of());
    }
}

class RoundRobinLoadBalancer {
    private final ServiceRegistry registry;
    private final AtomicInteger position = new AtomicInteger();

    RoundRobinLoadBalancer(ServiceRegistry registry) { this.registry = registry; }

    Instance choose(String serviceId) {
        List<Instance> list = registry.instances(serviceId);
        if (list.isEmpty()) throw new IllegalStateException("No instances for " + serviceId);
        int index = Math.floorMod(position.getAndIncrement(), list.size());
        return list.get(index);
    }
}

Output

Request 1 -> http://10.0.0.11:8081
Request 2 -> http://10.0.0.12:8081
Request 3 -> http://10.0.0.13:8081
Request 4 -> http://10.0.0.11:8081
Request 5 -> http://10.0.0.12:8081
Request 6 -> http://10.0.0.13:8081
--- 10.0.0.12 stops sending heartbeats and is removed ---
Request 7 -> http://10.0.0.11:8081
Request 8 -> http://10.0.0.13:8081
Request 9 -> http://10.0.0.11:8081
Request 10 -> http://10.0.0.13:8081

Tip: Spring's real RoundRobinLoadBalancer starts at a random position (so all callers do not hit instance 1 first) and uses an AtomicInteger exactly like this, so it is thread-safe.

Other Strategies

Strategy	How it picks	Use when
Round robin (default)	Next instance in turn	Instances are the same size
Random	`RandomLoadBalancer`	Many callers, simple spread
Weighted	`spring.cloud.loadbalancer.configurations: weighted` with instance metadata `weight`	Some machines are bigger
Zone preference	Same zone/data centre first	Multi-zone deployments, lower latency
Health check	Only instances whose `/actuator/health` is UP	No registry, static list of URLs

spring:
  cloud:
    loadbalancer:
      configurations: zone-preference   # or: weighted, health-check
      cache:
        ttl: 35s                        # how long the instance list is cached

Exercise

Exercise: Change RoundRobinLoadBalancer.choose so it skips instances whose host is in a Set<String> unhealthy (marked down by a health check), but still uses round robin among the rest.

Show answer

Instance choose(String serviceId, Set<String> unhealthy) {
    List<Instance> healthy = registry.instances(serviceId).stream()
            .filter(i -> !unhealthy.contains(i.host()))
            .toList();
    if (healthy.isEmpty()) throw new IllegalStateException("No healthy instances for " + serviceId);
    return healthy.get(Math.floorMod(position.getAndIncrement(), healthy.size()));
}