Consuming Events with KafkaListener

A Listener in payment-service

@Component
@RequiredArgsConstructor
@Slf4j
class OrderPlacedListener {

    private final PaymentService paymentService;

    @KafkaListener(topics = "order-placed", groupId = "payment-service", concurrency = "3")
    void onOrderPlaced(OrderPlacedEvent event,
                       @Header(KafkaHeaders.RECEIVED_PARTITION) int partition,
                       @Header(KafkaHeaders.OFFSET) long offset) {
        log.info("OrderPlaced {} (p{}@{}) amount {}", event.orderId(), partition, offset, event.totalAmount());
        paymentService.startPayment(event);
    }
}
spring:
  kafka:
    bootstrap-servers: ${KAFKA_SERVERS:localhost:9092}
    consumer:
      group-id: payment-service
      auto-offset-reset: earliest     # new group: start from the oldest record
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
      properties:
        spring.deserializer.value.delegate.class: org.springframework.kafka.support.serializer.JsonDeserializer
        spring.json.value.default.type: com.shopease.payment.events.OrderPlacedEvent
        spring.json.trusted.packages: "com.shopease.*"
  • concurrency = "3" starts 3 consumer threads, one per partition. More threads than partitions would sit idle.
  • ErrorHandlingDeserializer wraps the JSON deserializer so one broken message ("poison pill") does not crash the listener in a loop.
  • Spring commits the offset after the method returns normally (ack mode BATCH by default).
  • Each service has its own copy of the event class in its own package; they share the JSON contract, not a jar.

Consumer Groups in Action

Example

Two instances of payment-service share the 3 partitions; notification-service reads the same events independently:

docker exec kafka /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --all-groups

Example output

GROUP                 TOPIC         PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID
notification-service  order-placed  0          12              12              0    consumer-notification-service-1-4f1c...
notification-service  order-placed  1          9               9               0    consumer-notification-service-1-4f1c...
notification-service  order-placed  2          18              18              0    consumer-notification-service-1-4f1c...
payment-service       order-placed  0          12              12              0    consumer-payment-service-1-a9d2...
payment-service       order-placed  1          9               9               0    consumer-payment-service-1-a9d2...
payment-service       order-placed  2          16              18              2    consumer-payment-service-2-c07e...

Tip: LAG = records written but not yet processed by that group. Rising lag means consumers are too slow: add instances (up to the partition count) or make processing faster. Alert on it.