Careeroza

Retry with Backoff

Retry Short Glitches

Many failures last milliseconds: a pod restarting, a dropped connection. Retrying fixes them. Rules:

  • Retry only transient errors (timeouts, 502/503). Never retry 400/404/409.
  • Wait longer each time (exponential backoff: 500 ms, 1 s, 2 s) so you do not hammer a sick service. Add jitter (randomness) in production so 1000 clients do not retry at the same moment.
  • Retry only idempotent operations (GET, or POST with an idempotency key). Retrying a plain "charge ₹2,998" may charge twice.

Example

Retry with exponential backoff. Waits are added up instead of really sleeping:

import java.util.function.IntFunction;

public class Main {
    public static void main(String[] args) {
        Retry retry = new Retry(4, 500, 2.0);

        System.out.println("1) product-service recovers on attempt 3");
        System.out.println("   result: " + retry.execute(attempt -> {
            if (attempt < 3) throw new ServiceUnavailable("503 Service Unavailable");
            return "boAt Rockerz 450";
        }));

        System.out.println("2) product-service stays down");
        try {
            retry.execute(attempt -> { throw new ServiceUnavailable("connect timed out"); });
        } catch (ServiceUnavailable e) {
            System.out.println("   fallback: show cached product (" + e.getMessage() + ")");
        }

        System.out.println("3) bad request is not retried");
        try {
            retry.execute(attempt -> { throw new BadRequest("400 quantity must be > 0"); });
        } catch (BadRequest e) {
            System.out.println("   error returned to client: " + e.getMessage());
        }
    }
}

class ServiceUnavailable extends RuntimeException { ServiceUnavailable(String m) { super(m); } }
class BadRequest extends RuntimeException { BadRequest(String m) { super(m); } }

class Retry {
    private final int maxAttempts;
    private final long initialWaitMs;
    private final double multiplier;

    Retry(int maxAttempts, long initialWaitMs, double multiplier) {
        this.maxAttempts = maxAttempts;
        this.initialWaitMs = initialWaitMs;
        this.multiplier = multiplier;
    }

    String execute(IntFunction<String> call) {
        long totalWait = 0;
        for (int attempt = 1; ; attempt++) {
            try {
                String result = call.apply(attempt);
                System.out.println("   attempt " + attempt + ": success (waited " + totalWait + " ms in total)");
                return result;
            } catch (ServiceUnavailable e) {
                if (attempt == maxAttempts) {
                    System.out.println("   attempt " + attempt + ": " + e.getMessage() + " -> giving up");
                    throw e;
                }
                long wait = (long) (initialWaitMs * Math.pow(multiplier, attempt - 1));
                totalWait += wait;
                System.out.println("   attempt " + attempt + ": " + e.getMessage() + " -> wait " + wait + " ms");
            } catch (BadRequest e) {
                System.out.println("   attempt " + attempt + ": " + e.getMessage() + " -> not retryable");
                throw e;
            }
        }
    }
}

Output

1) product-service recovers on attempt 3
   attempt 1: 503 Service Unavailable -> wait 500 ms
   attempt 2: 503 Service Unavailable -> wait 1000 ms
   attempt 3: success (waited 1500 ms in total)
   result: boAt Rockerz 450
2) product-service stays down
   attempt 1: connect timed out -> wait 500 ms
   attempt 2: connect timed out -> wait 1000 ms
   attempt 3: connect timed out -> wait 2000 ms
   attempt 4: connect timed out -> giving up
   fallback: show cached product (connect timed out)
3) bad request is not retried
   attempt 1: 400 quantity must be > 0 -> not retryable
   error returned to client: 400 quantity must be > 0

Retry in Spring Boot

@Retry(name = "productService", fallbackMethod = "cachedProduct")
@CircuitBreaker(name = "productService")
public ProductDto getProduct(Long id) {
    return productClient.getProduct(id);
}
resilience4j:
  retry:
    instances:
      productService:
        max-attempts: 3                      # 1 call + 2 retries
        wait-duration: 500ms
        enable-exponential-backoff: true
        exponential-backoff-multiplier: 2    # 500ms, 1s
        retry-exceptions:
          - java.io.IOException
          - feign.RetryableException
          - com.shopease.order.client.ServiceUnavailableException
        ignore-exceptions:
          - io.github.resilience4j.circuitbreaker.CallNotPermittedException  # circuit open: do not retry

Tip: Watch the multiplication: gateway retries 2× → order-service retries 3× → Feign retries 3× = up to 18 calls to product-service for one click. Retry at one layer only (usually the caller closest to the failing service).