Idempotency

Why Duplicates Happen

An operation is idempotent when doing it twice has the same effect as doing it once. In distributed systems duplicates are normal, not rare:

  • A client times out and retries a POST /payments that actually succeeded.
  • Resilience4j Retry or the gateway retries a call after a slow response.
  • Kafka redelivers a message when a consumer crashes before committing its offset (at-least-once).
  • The outbox relay re-sends a row after a crash.
  • A user double-taps the "Pay ₹2,998" button on a slow 3G connection.
HTTP methodIdempotent by definition?Note
GET, HEADYesRead only
PUTYesSets a full value; repeating gives the same state
DELETEYesSecond call finds nothing to delete
POSTNoNeeds an idempotency key to be safe to retry
PATCHDepends"set stock=5" yes, "add 1 to stock" no

Example: Idempotent Payment with an Idempotency Key

Example

The client generates a unique key per payment attempt and sends it in an Idempotency-Key header. The server stores the first response for that key and replays it for retries. Reusing a key with a different body is rejected. The card is charged only twice for two real payments, even though four requests arrive:

import java.util.*;

public class Main {
    public static void main(String[] args) {
        CardGateway gateway = new CardGateway();
        PaymentService service = new PaymentService(gateway);

        String key = "7f3c9a2e-order-1001";
        System.out.println(service.pay(key, new PayRequest(1001, 2998)));
        System.out.println("(client timed out before seeing the response, so it retries)");
        System.out.println(service.pay(key, new PayRequest(1001, 2998)));
        System.out.println(service.pay(key, new PayRequest(1001, 5000)));   // same key, different body
        System.out.println(service.pay("a91d44b0-order-1002", new PayRequest(1002, 1499)));
        System.out.println("card charged " + gateway.charges + " times, total Rs." + gateway.total);
    }
}

record PayRequest(long orderId, int amount) {}
record Response(int status, String body) {
    public String toString() { return status + " " + body; }
}
record StoredResult(PayRequest request, Response response) {}

class CardGateway {
    int charges = 0, total = 0;
    String charge(int amount) { charges++; total += amount; return "pay_" + (1000 + charges); }
}

class PaymentService {
    private final Map<String, StoredResult> results = new LinkedHashMap<>();   // a DB table in real life
    private final CardGateway gateway;
    PaymentService(CardGateway gateway) { this.gateway = gateway; }

    Response pay(String idempotencyKey, PayRequest req) {
        StoredResult previous = results.get(idempotencyKey);
        if (previous != null) {
            if (!previous.request().equals(req))
                return new Response(422, "{error: idempotency key reused with a different request}");
            return new Response(previous.response().status(), previous.response().body() + "  [replayed]");
        }
        String paymentId = gateway.charge(req.amount());
        Response response = new Response(201, "{paymentId: " + paymentId + ", orderId: " + req.orderId()
                + ", amount: " + req.amount() + "}");
        results.put(idempotencyKey, new StoredResult(req, response));
        return response;
    }
}

Output

201 {paymentId: pay_1001, orderId: 1001, amount: 2998}
(client timed out before seeing the response, so it retries)
201 {paymentId: pay_1001, orderId: 1001, amount: 2998}  [replayed]
422 {error: idempotency key reused with a different request}
201 {paymentId: pay_1002, orderId: 1002, amount: 1499}
card charged 2 times, total Rs.4497

Idempotency Keys in Spring Boot

@Entity
@Table(name = "idempotency_record",
       uniqueConstraints = @UniqueConstraint(columnNames = "idempotency_key"))
public class IdempotencyRecord {
    @Id @GeneratedValue private Long id;
    @Column(name = "idempotency_key", nullable = false) private String key;
    private String requestHash;          // SHA-256 of the request body
    private int responseStatus;
    @Lob private String responseBody;
    private Instant createdAt;           // purge after 24 hours
    // constructor, getters
}

@RestController
@RequestMapping("/api/payments")
@RequiredArgsConstructor
public class PaymentController {
    private final IdempotencyRecordRepository records;
    private final PaymentService payments;
    private final ObjectMapper json;

    @PostMapping
    @Transactional
    public ResponseEntity<String> pay(@RequestHeader("Idempotency-Key") String key,
                                      @Valid @RequestBody PaymentRequest req) throws Exception {
        String hash = Hashing.sha256(json.writeValueAsString(req));
        Optional<IdempotencyRecord> existing = records.findByKey(key);
        if (existing.isPresent()) {
            IdempotencyRecord r = existing.get();
            if (!r.getRequestHash().equals(hash)) {
                return ResponseEntity.unprocessableEntity().body("{\"error\":\"key reused with a different body\"}");
            }
            return ResponseEntity.status(r.getResponseStatus()).header("Idempotent-Replayed", "true").body(r.getResponseBody());
        }
        PaymentResponse result = payments.charge(req);   // calls Razorpay with the same key
        String body = json.writeValueAsString(result);
        records.save(new IdempotencyRecord(key, hash, 201, body));   // unique constraint stops a parallel duplicate
        return ResponseEntity.status(201).body(body);
    }
}

Example

The second call with the same key does not charge again:

curl -X POST localhost:8084/api/payments \
  -H "Idempotency-Key: 7f3c9a2e-order-1001" -H "Content-Type: application/json" \
  -d '{"orderId": 1001, "amount": 2998.00, "method": "UPI"}'

# same command again (client retry)

Example output

HTTP/1.1 201 Created
{"paymentId":"pay_NqX81kLm2","orderId":1001,"amount":2998.00,"status":"CAPTURED"}

HTTP/1.1 201 Created
Idempotent-Replayed: true
{"paymentId":"pay_NqX81kLm2","orderId":1001,"amount":2998.00,"status":"CAPTURED"}

Note: Two retries can arrive at the same millisecond. The unique constraint on the key makes the second insert fail, so only one charge wins. Some teams first insert an IN_PROGRESS row and return 409 Conflict to a parallel duplicate.

Idempotent Kafka Consumers

For events, use the event ID as the key: record every processed event ID in the same transaction as the business change, and skip IDs you have already seen.

@KafkaListener(topics = "order-events", groupId = "inventory-service")
@Transactional
public void onOrderPlaced(OrderPlaced event) {
    if (processedEvents.existsById(event.eventId())) {
        log.info("Duplicate event {} ignored", event.eventId());
        return;
    }
    inventoryService.reserve(event.orderId(), event.items());
    processedEvents.save(new ProcessedEvent(event.eventId(), Instant.now()));  // PK = eventId
}

Tip: Natural idempotency is even better when possible: "set order 1001 status to PAID" is safe to repeat; "add ₹2,998 to the wallet" is not. Prefer absolute updates over increments, or guard increments with a processed-events table.

Exercise

Exercise: In the idempotent payment example, keys stay in memory forever. Change StoredResult to also store a createdAtMinute (an int), and make pay treat a key older than 1440 minutes (24 h) as new. Pass the current minute as a parameter so the program stays deterministic.

Show answer
record StoredResult(PayRequest request, Response response, int createdAtMinute) {}

Response pay(String key, PayRequest req, int nowMinute) {
    StoredResult previous = results.get(key);
    if (previous != null && nowMinute - previous.createdAtMinute() > 1440) {
        results.remove(key);          // expired: forget it
        previous = null;
    }
    if (previous != null) { /* same replay / 422 logic as before */ }
    String paymentId = gateway.charge(req.amount());
    Response response = new Response(201, "{paymentId: " + paymentId + "}");
    results.put(key, new StoredResult(req, response, nowMinute));
    return response;
}