Resilience Questions

A circuit breaker wraps calls to a dependency and fails fast when the dependency is unhealthy, so threads are not wasted waiting and the failing service gets time to recover.

failure rate >= threshold
 CLOSED ───────────────────────────► OPEN  (calls rejected, fallback runs)
   ▲                                   │
   │ trial calls succeed               │ wait-duration-in-open-state elapses
   │                                   ▼
   └──────────────────────────── HALF_OPEN (allow N trial calls)
                trial calls fail ──► back to OPEN

Resilience4j measures failure rate (and slow-call rate) over a sliding window of the last N calls or N seconds. See the Coding Round for a runnable implementation.

@Service
public class PaymentGateway {
    @CircuitBreaker(name = "payment", fallbackMethod = "payLater")
    @Retry(name = "payment")
    public PaymentResult charge(ChargeRequest req) {
        return paymentClient.charge(req);
    }

    // same parameters + the exception
    private PaymentResult payLater(ChargeRequest req, Throwable ex) {
        return PaymentResult.pending(req.orderId(), "Payment provider busy, we will retry");
    }
}

resilience4j:
  circuitbreaker:
    instances:
      payment:
        sliding-window-size: 20
        minimum-number-of-calls: 10
        failure-rate-threshold: 50
        slow-call-duration-threshold: 2s
        slow-call-rate-threshold: 60
        wait-duration-in-open-state: 30s
        permitted-number-of-calls-in-half-open-state: 3
  retry:
    instances:
      payment:
        max-attempts: 3
        wait-duration: 500ms
        enable-exponential-backoff: true
        retry-exceptions: [java.io.IOException, java.util.concurrent.TimeoutException]

Yes. Retries help with transient errors (a dropped connection). But if a service is overloaded, every client retrying 3 times multiplies traffic by 3–4x (retry storm) and keeps it down. Safe retries need:

• exponential backoff with jitter, a small max-attempts, and a total timeout budget;
• retry only idempotent operations (GET, or POST with an idempotency key) and only retryable errors (5xx, timeouts, not 400/404);
• a circuit breaker so retries stop when the dependency is clearly down.

Example: payment-service calls a bank API limited to 100 req/s. A rate limiter keeps us under the quota; a bulkhead of 20 concurrent calls means a slow bank cannot exhaust Tomcat's 200 threads, so product browsing keeps working.

• Return cached or default data: last known product price, "recommendations unavailable", a static top-10 list.
• Degrade the feature, not the page: hide the reviews widget instead of failing the product page.
• Queue the work for later: accept the order as PAYMENT_PENDING and retry asynchronously.
• Do not fake success for critical operations (never return "payment ok" from a fallback). Log and emit a metric when a fallback fires.

One slow service (payment) makes its callers (order) hold threads while waiting; their thread pools fill, so they become slow too, and the failure spreads up to the gateway. Prevent it with timeouts on every call, circuit breakers, bulkheads, load shedding (return 503 early), async messaging for non-critical steps, and autoscaling with health-based routing.