Careeroza

Metrics with Prometheus and Grafana

Expose Metrics from Each Service

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

Configuration

Actuator then serves all metrics in Prometheus text format:

application.yml

management:
  endpoints:
    web:
      exposure:
        include: health,info,prometheus
  metrics:
    tags:
      application: order-service           # label added to every metric
    distribution:
      percentiles-histogram:
        http.server.requests: true         # enables p95/p99 in Prometheus

Example output

$ curl -s localhost:8082/actuator/prometheus | grep -E "http_server_requests_seconds_count|hikaricp_connections_active"
http_server_requests_seconds_count{application="order-service",method="POST",outcome="SUCCESS",status="201",uri="/api/orders"} 1532
http_server_requests_seconds_count{application="order-service",method="POST",outcome="SERVER_ERROR",status="500",uri="/api/orders"} 7
hikaricp_connections_active{application="order-service",pool="HikariPool-1"} 3

Custom Business Metrics

@Service
public class OrderMetrics {
    private final Counter ordersPlaced;
    private final DistributionSummary orderValue;
    private final MeterRegistry registry;

    public OrderMetrics(MeterRegistry registry) {
        this.registry = registry;
        this.ordersPlaced = Counter.builder("shopease.orders.placed")
                .description("Orders successfully placed").register(registry);
        this.orderValue = DistributionSummary.builder("shopease.order.value")
                .baseUnit("rupees").publishPercentiles(0.5, 0.95).register(registry);
    }

    public void placed(Order order) {
        ordersPlaced.increment();
        orderValue.record(order.getTotal().doubleValue());
    }

    public void paymentFailed(String reason) {             // reason: INSUFFICIENT_FUNDS, TIMEOUT, ...
        registry.counter("shopease.payments.failed", "reason", reason).increment();
    }
}

Note: Tags (labels) must have low cardinality: reason, status, region are fine; userId or orderId would create millions of time series and crash Prometheus.

Prometheus Scrapes, Grafana Shows

Prometheus configuration

prometheus.yml

global:
  scrape_interval: 15s
scrape_configs:
  - job_name: shopease
    metrics_path: /actuator/prometheus
    static_configs:
      - targets: ["order-service:8082", "payment-service:8084", "inventory-service:8083", "api-gateway:8080"]
QuestionPromQL
Requests per second per servicesum by (application) (rate(http_server_requests_seconds_count[5m]))
Error rate of order-servicesum(rate(http_server_requests_seconds_count{application="order-service",outcome="SERVER_ERROR"}[5m])) / sum(rate(http_server_requests_seconds_count{application="order-service"}[5m]))
p95 latencyhistogram_quantile(0.95, sum by (le) (rate(http_server_requests_seconds_bucket{application="order-service"}[5m])))
Orders per minuterate(shopease_orders_placed_total[5m]) * 60
Circuit breaker open?resilience4j_circuitbreaker_state{state="open"} == 1

Tip: Import ready-made Grafana dashboards: "JVM (Micrometer)" and "Spring Boot 3.x Statistics" from grafana.com. In Kubernetes, the Prometheus Operator discovers pods through a ServiceMonitor instead of a static target list.

Alerting

Alert rule

An alert rule that pages the on-call engineer when checkout errors spike:

alerts.yml

groups:
  - name: shopease-alerts
    rules:
      - alert: OrderServiceHighErrorRate
        expr: |
          sum(rate(http_server_requests_seconds_count{application="order-service",outcome="SERVER_ERROR"}[5m]))
            / sum(rate(http_server_requests_seconds_count{application="order-service"}[5m])) > 0.01
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "order-service 5xx rate above 1% for 5 minutes"

Example output

[FIRING:1] OrderServiceHighErrorRate (severity=critical)
summary: order-service 5xx rate above 1% for 5 minutes
-> Slack #shopease-oncall, PagerDuty