Monitoring Basics

Monitoring & Observability: The "Eyes" of the System

When a microservice-based system fails, it rarely manifests as a clear error message. It usually presents as "latency" or "some users are seeing intermittent 500 errors." Observability is the ability to understand the internal state of your system by analyzing its external outputs. This is achieved through the "Three Pillars": Logging, Metrics, and Tracing.

1. Logging

Logs are immutable, timestamped records of discrete events that happened within the system (e.g., "User logged in," "Database connection timed out").

• Structure: Logs should always be structured (typically in JSON format) so they can be indexed and queried by machines.

• Centralization: In a distributed system, you cannot ssh into 50 servers to grep logs. You must ship all logs to a centralized stack (like ELK—Elasticsearch, Logstash, Kibana) where you can perform full-text searches across your entire cluster.

• Levels: Use standardized levels (DEBUG, INFO, WARN, ERROR) to filter noise from actual issues.

2. Metrics

Metrics are numerical representations of data measured over intervals of time. They are perfect for answering questions like "How much CPU is my service using?" or "How many requests per second are we handling?"

• Types:

  • Gauges: A snapshot of a value at a point in time (e.g., current memory usage).

  • Counters: A cumulative value that only ever increases (e.g., total number of HTTP requests).

  • Histograms/Summaries: Useful for tracking distribution (e.g., 99th percentile latency—how slow is the request for the slowest 1% of users?).

• Tooling: Prometheus is the industry standard for collecting metrics, while Grafana is the standard for visualizing them in dashboards.

3. Tracing

While logs tell you what happened and metrics tell you how often, Tracing tells you where the time went.

• The Request Lifecycle: Tracing follows a single user request as it traverses through various microservices. By attaching a trace_id to the request header, you can build a visualization of the call chain.

• The Value: Tracing is the only way to effectively debug "cascading latency"—where Service A is slow only because Service C (three hops away) is experiencing database contention.

4. Alerting Systems

Monitoring is useless if nobody is notified when things go wrong. Alerting is the "automated nervous system" that informs engineers of production issues.

• Threshold-Based Alerts: "If CPU > 90% for 5 minutes, page the On-Call Engineer."

• Symptom-Based Alerts (SLIs/SLOs): Modern SRE (Site Reliability Engineering) principles suggest alerting on symptoms (what the user experiences) rather than causes (server CPU).

  • Example: Don't alert if a specific server is down; alert if the Error Rate for the "Checkout Service" exceeds 1% of total traffic.

• Alert Fatigue: The fastest way to destroy an engineering team's productivity is to spam them with "low priority" alerts. Only alert on issues that require human intervention.

Monitoring Reference Matrix