Careeroza

Real World Example

Sale-Day Monitoring

On sale days, e-commerce teams watch a "war room" dashboard: orders per minute, payment success rate, p95 checkout latency and DB pool usage. Alerts fire before customers start complaining on social media.

Example

An alert rule for ShopEase payments:

alerts.yml

groups:
  - name: shopease
    rules:
      - alert: PaymentFailureRateHigh
        expr: |
          sum(rate(shopease_payment_seconds_count{outcome="failed"}[5m]))
            / sum(rate(shopease_payment_seconds_count[5m])) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "More than 5% of payments are failing"
      - alert: CheckoutSlow
        expr: histogram_quantile(0.95, sum by (le) (rate(http_server_requests_seconds_bucket{uri="/api/checkout"}[5m]))) > 1.5
        for: 10m
        labels:
          severity: warning
// tag the timer with the outcome so the alert can split success and failure
public PaymentResult charge(Cart cart) {
    Timer.Sample sample = Timer.start(registry);
    PaymentResult result = gateway.charge(cart);
    sample.stop(registry.timer("shopease.payment", "outcome", result.success() ? "success" : "failed"));
    return result;
}

Example

What the on-call engineer receives in Slack (channel #shopease-oncall):

Alertmanager -> Slack webhook

Example output

[FIRING:1] PaymentFailureRateHigh (severity=critical)
summary: More than 5% of payments are failing
value: 0.083  (8.3% over the last 5 minutes)
dashboard: https://grafana.shopease.in/d/checkout