System Design and Scenario Questions

client ─POST /api/orders (Idempotency-Key)─► gateway ─► order-service
  1. order-service: validate cart, price snapshot from product-service (sync)
  2. save Order(PENDING) + outbox "OrderCreated" (one local tx)  ──► 202 Accepted {orderId}
  3. inventory-service  on OrderCreated  → reserve stock (TTL 15 min) → "StockReserved"
                                           or "StockRejected" ──► order CANCELLED
  4. payment-service    on StockReserved → charge with key=orderId → "PaymentCompleted"
                                           or "PaymentFailed"  ──► release stock, order CANCELLED
  5. order-service      on PaymentCompleted → order CONFIRMED → "OrderConfirmed"
  6. notification-service on OrderConfirmed → email/SMS

• Choreography saga, outbox in every service, idempotent consumers keyed by event id, orderId as the Kafka key so events of one order stay ordered.
• Client sees PENDING and gets the final status via polling or WebSocket. A timeout job cancels orders stuck in PENDING.

1. Contain: timeouts on the caller (e.g. 2 s), circuit breaker opens on slow-call rate, bulkhead so other features keep working; move payment to async (order PENDING, pay later) if possible.
2. Find: Grafana for latency/error rate per endpoint; Zipkin trace of a slow request to see which span is slow (DB query? the bank API? GC pauses?); check thread pool, HikariCP pool wait time, CPU, GC logs.
3. Fix the cause: add an index, increase pool size, scale out pods, add caching for read calls, or rate-limit/queue calls to the external provider.
4. Prevent: SLO alerts on p99, load tests before sales (Big Billion Days style traffic).

• Client sends an Idempotency-Key per checkout attempt; the Pay button is disabled after the first click.
• payment-service stores the key with a unique constraint before calling the provider; a duplicate request returns the stored result.
• Pass the same key (or orderId) to the payment provider (Razorpay/Stripe support idempotency keys or a unique receipt id), so retries after a timeout do not create a second charge.
• On timeout, do not blindly retry: query the provider for the status of that order id first. Reconcile daily with the provider's settlement report.
• Kafka consumers dedupe by event id; one payment per orderId enforced by a unique index.

• Check consumer lag per partition (Kafka UI, Burrow, kafka_consumergroup_lag metric).
• Scale consumers up to the number of partitions; if already equal, increase partitions (note: changes key → partition mapping for new messages).
• Make processing faster: batch listener, bulk DB writes, avoid a synchronous HTTP call per message, raise concurrency on the listener container.
• If one partition is hot (one key gets most traffic), choose a better key.

• Run with -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dumps, analyse with Eclipse MAT: look at the dominator tree.
• Common causes: unbounded caches/maps, loading whole tables into memory (no pagination), large Kafka batches, thread leaks, container memory limit lower than -Xmx + metaspace + thread stacks.
• Use -XX:MaxRAMPercentage=75 in containers instead of a fixed -Xmx; watch heap metrics in Grafana to see the growth pattern.

• External: the gateway validates the user's JWT (OAuth2 resource server) and relays the token (TokenRelay filter) so downstream services can authorise by user and roles.
• Service-to-service: OAuth2 client credentials tokens with scopes, or mTLS via a service mesh (Istio, Linkerd).
• Every service still validates tokens (zero trust); do not trust "internal network" alone. See the Spring Security material for details.

Use a token bucket per key (user id, API key or IP): capacity = burst, refill rate = allowed RPS. Store buckets in Redis so all gateway instances share them, and update atomically with a Lua script (Spring Cloud Gateway's RedisRateLimiter does exactly this). Return 429 Too Many Requests with X-RateLimit-Remaining and Retry-After headers. Choose fail-open (allow if Redis is down) for normal APIs and fail-closed for sensitive ones like OTP/login. Runnable token bucket in the Coding Round.