Failure Scenarios

What Can Go Wrong and How ShopEase Handles It

Failure	What the user sees	How it is handled
App retries `POST /api/orders` after a timeout	One order only	Idempotency key with a unique constraint; the stored response is replayed
product-service is slow or down	"Unable to price your cart, try again" (fast)	Timeout 800 ms + circuit breaker; nothing is saved, so nothing to undo
Not enough stock	Order cancelled: "Item went out of stock"	StockRejected → saga cancels the order; no payment was taken
Payment declined	Order cancelled, stock back on sale	PaymentDeclined → compensation ReleaseStock → CANCELLED
Kafka down for 5 minutes	Order stays PENDING a bit longer	Events wait in the outbox tables; the relay sends them when Kafka is back
order-service Pod crashes mid-saga	Nothing visible	Saga state is in the DB; another Pod continues on the next reply; idempotent handlers absorb redelivery
payment-service never replies	Order cancelled after 15 min, refund if charged	Saga timeout job compensates stuck sagas; Razorpay reconciliation job checks real status
Duplicate Kafka message	Nothing visible	processed_events table in each consumer
Poison message (cannot be parsed)	Nothing visible	3 retries with backoff, then dead-letter topic + alert
Bad release of payment-service	A few users see errors for a minute	Canary analysis aborts the rollout automatically
A whole availability zone goes down	Slightly slower responses	Pods spread across zones, PodDisruptionBudgets, replicated databases

Saga Timeouts

@Component
@RequiredArgsConstructor
@Slf4j
public class StuckSagaSweeper {
    private final OrderSagaRepository sagas;
    private final OrderSagaOrchestrator orchestrator;

    @Scheduled(fixedDelay = 60_000)
    @SchedulerLock(name = "stuck-saga-sweeper")          // ShedLock: one Pod runs it at a time
    @Transactional
    public void compensateStuckSagas() {
        Instant cutoff = Instant.now().minus(Duration.ofMinutes(15));
        for (OrderSaga saga : sagas.findTop100ByStatusAndUpdatedAtBefore(SagaStatus.RUNNING, cutoff)) {
            log.warn("Saga for order {} stuck at {} since {}; compensating", saga.getOrderId(), saga.getStep(), saga.getUpdatedAt());
            orchestrator.compensate(saga, "TIMEOUT");     // refund (idempotent) + release stock + cancel
        }
    }
}

Tip: A timeout does not mean the payment failed. It may have succeeded slowly. That is why the compensation is an idempotent refund-if-charged, and a reconciliation job compares ShopEase payments with the Razorpay settlement report every night.

Retries and Dead-Letter Topics

@Configuration
public class KafkaErrorHandling {
    @Bean
    DefaultErrorHandler kafkaErrorHandler(KafkaTemplate<Object, Object> template) {
        var recoverer = new DeadLetterPublishingRecoverer(template);         // sends to <topic>.DLT
        var handler = new DefaultErrorHandler(recoverer, new FixedBackOff(1_000L, 3));   // 3 retries, 1 s apart
        handler.addNotRetryableExceptions(DeserializationException.class, ValidationException.class);
        return handler;                                                       // picked up by Spring Boot
    }
}

Example

kubectl -n shopease logs deploy/inventory-service | grep -i "dlt"

Example output

WARN 1 --- [inventory-service] [ntainer#0-0-C-1] [7a1c...-3e9f...] o.s.k.l.DeadLetterPublishingRecoverer : Record order-events-4@18233 sent to order-events.DLT after 4 attempts: ValidationException: quantity must be positive

Exercise

Exercise: Customer support reports: "Order 1077 was charged but shows CANCELLED." Using the tools from this level, list the steps to find out what happened.

Show answer

1. Find the trace: search the logs (Loki/Kibana) for orderId=1077 and copy its traceId.
2. Open the trace in Zipkin/Tempo: see which saga steps ran and how long each took.
3. Check saga_instance for order 1077: step, status, updatedAt.
   e.g. step=CHARGING, status=CANCELLED, reason=TIMEOUT -> payment replied after the 15 min timeout.
4. Check payment-service: payments + refunds for order 1077, and the Razorpay dashboard.
5. If the refund failed or was never sent: look in payment-commands.DLT and the error logs.
6. Fix the data (trigger the idempotent refund), then fix the cause:
   e.g. raise the timeout, make the late PaymentCompleted reply trigger an automatic refund,
   add an alert on "charged but cancelled" orders.