Careeroza

Production Checklist

Before Going Live

AreaChecklist
DesignEach service owns its data; no shared tables; APIs and events documented (OpenAPI, AsyncAPI)
Data consistencySaga with compensations for every multi-service write; outbox for every event; idempotent handlers
ResilienceTimeouts on every remote call; circuit breakers; retries only for idempotent operations; fallbacks for optional data
SecurityJWT validation in every service; audience and scope checks; client credentials or mTLS between services; no secrets in Git
ObservabilityTrace ID in every log line; RED metrics dashboards per service; alerts with runbooks; DLT monitoring
KubernetesResource requests/limits; readiness, liveness and startup probes; graceful shutdown; PodDisruptionBudgets; HPA
DeliveryOne pipeline per service; immutable image tags; expand/contract migrations; canary with automatic analysis; one-command rollback
OperationsBackups tested by restoring; load test at 3x expected peak; chaos test (kill a Pod, stop Kafka) in staging

Load Test Before a Sale

Example

A quick load test with k6 against staging shows whether the system meets its targets:

k6 run --vus 500 --duration 5m checkout-test.js

Example output

     checks.........................: 99.87% ✓ 148203  ✗ 192
     http_req_duration..............: avg=142ms  p(95)=388ms  p(99)=712ms
     http_reqs......................: 148395  494.6/s
     orders_confirmed...............: 49421
     orders_cancelled_out_of_stock..: 311

     ✓ p(95) < 500ms   ✓ error rate < 1%

Note: If the numbers are bad, use the traces to find the slowest span first. In practice the culprit is usually a missing index, an N+1 call pattern, a too-small connection pool, or a synchronous call that should be an event.