Careeroza

Real World Example

How Large Teams Ship Many Times a Day

Companies like Netflix, Amazon and Swiggy deploy hundreds of times a day. No human checks each release by hand; the pipeline does:

  1. Developer merges a pull request to main after review and green checks.
  2. CI runs tests and contract tests, builds an image tagged with the Git SHA, scans it.
  3. GitOps: CI commits the new tag to the deployment repo; Argo CD applies it to staging, then production.
  4. Production uses a canary: 5% of traffic, automatic analysis of error rate and p95 latency against the stable version (Netflix calls this automated canary analysis with Kayenta/Spinnaker).
  5. Bad metrics roll back automatically and alert the team in Slack. Good metrics promote to 100%.

An Analysis Template for the Canary

AnalysisTemplate

k8s/order-service/analysis.yaml

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: order-error-rate
  namespace: shopease
spec:
  metrics:
    - name: error-rate
      interval: 1m
      count: 5
      failureLimit: 1
      successCondition: result[0] < 0.01
      provider:
        prometheus:
          address: http://prometheus.observability:9090
          query: |
            sum(rate(http_server_requests_seconds_count{application="order-service",outcome="SERVER_ERROR"}[2m]))
            /
            sum(rate(http_server_requests_seconds_count{application="order-service"}[2m]))

Example output

$ kubectl argo rollouts get rollout order-service -n shopease
Status:          ✖ Degraded
Message:         RolloutAborted: Rollout aborted update to revision 14: Metric "error-rate" assessed Failed due to failed (2) > failureLimit (1)
Images:          ghcr.io/shopease/order-service:2.0.4 (stable)

Tip: The metrics you built in the Observability module are what make automatic canaries possible. Without good metrics, every deployment needs a human watching dashboards.