Careeroza

Errors, Retries and Dead Letter Topics

When the Listener Throws

If onOrderPlaced throws, Spring Kafka's DefaultErrorHandler retries the record (by default 9 more times with no wait) and then logs and skips it. For production, back off between retries and send the failed record to a dead letter topic (DLT) so nothing is lost:

@Configuration
class KafkaErrorConfig {

    @Bean
    DefaultErrorHandler kafkaErrorHandler(KafkaTemplate<?, ?> template) {
        // after retries: publish to "order-placed.DLT", same partition
        DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(template);

        ExponentialBackOffWithMaxRetries backOff = new ExponentialBackOffWithMaxRetries(3);
        backOff.setInitialInterval(1_000);   // 1s, 2s, 4s
        backOff.setMultiplier(2.0);

        DefaultErrorHandler handler = new DefaultErrorHandler(recoverer, backOff);
        handler.addNotRetryableExceptions(InvalidOrderEventException.class);  // straight to DLT
        return handler;
    }
}

Example

Spring Boot wires a single CommonErrorHandler bean into the listener containers automatically:

# payment-service log when the bank client keeps failing

Example output

ERROR o.s.k.l.KafkaMessageListenerContainer : Error handler threw an exception ... BankTimeoutException  (attempt 1, retry in 1000 ms)
ERROR o.s.k.l.KafkaMessageListenerContainer : ... BankTimeoutException  (attempt 2, retry in 2000 ms)
ERROR o.s.k.l.KafkaMessageListenerContainer : ... BankTimeoutException  (attempt 3, retry in 4000 ms)
INFO  c.s.payment.DltMonitor : order-placed-2@17 moved to order-placed.DLT (orderId=1001)

Note: Blocking retries hold up the whole partition. For long waits use @RetryableTopic, which moves the record to retry topics (order-placed-retry-0, ...) so later records keep flowing.

Idempotent Consumers

Kafka gives at-least-once delivery: after a crash or rebalance, the same record can be delivered again. The consumer must make a duplicate harmless. Simplest way: remember processed eventIds in the same database transaction as the work:

@Transactional
public void startPayment(OrderPlacedEvent event) {
    if (processedEvents.existsById(event.eventId())) {
        log.info("Duplicate event {} ignored", event.eventId());
        return;
    }
    payments.save(Payment.pending(event.orderId(), event.totalAmount()));
    processedEvents.save(new ProcessedEvent(event.eventId(), Instant.now()));
}

Tip: Saga, outbox and idempotency in depth are in the advance level ("Distributed Data").