Blue-Green and Canary Deployments

Service

k8s/order-service/service.yaml

apiVersion: v1
kind: Service
metadata:
  name: order-service
  namespace: shopease
spec:
  selector:
    app: order-service
    color: blue              # change to green to switch ALL traffic
  ports:
    - port: 80
      targetPort: 8082

Example output

# green is deployed and tested on its own internal Service (order-service-green)
$ kubectl -n shopease patch service order-service \
    -p '{"spec":{"selector":{"app":"order-service","color":"green"}}}'
service/order-service patched

# problem? switch back in seconds
$ kubectl -n shopease patch service order-service \
    -p '{"spec":{"selector":{"app":"order-service","color":"blue"}}}'

Example

Each user is put in a bucket 0–99 from a stable hash of the user ID, so the same user always sees the same version (sticky). Traffic to the canary grows 10% → 25% → 50% → 100% while its error rate is checked after each step. The healthy build is promoted; the buggy one is aborted at 10%, so only about 1 in 10 users ever saw it:

public class Main {
    public static void main(String[] args) {
        canary("2.1.0", 250);   // healthy build: 1 error per 250 requests
        canary("2.2.0", 16);    // buggy build: 1 error per 16 requests
    }

    static void canary(String version, int canaryErrorEvery) {
        System.out.println("== canary order-service " + version + " ==");
        for (int weight : new int[] {10, 25, 50, 100}) {
            int[] total = new int[2], errors = new int[2];          // [0] stable, [1] canary
            for (int i = 1; i <= 2000; i++) {
                int bucket = Math.floorMod(("user-" + i).hashCode(), 100);   // sticky per user
                int v = bucket < weight ? 1 : 0;
                total[v]++;
                if (total[v] % (v == 1 ? canaryErrorEvery : 200) == 0) errors[v]++;
            }
            double stable = total[0] == 0 ? 0 : 100.0 * errors[0] / total[0];
            double canary = 100.0 * errors[1] / total[1];
            System.out.printf("weight %3d%% | stable %4d req %.2f%% err | canary %4d req %.2f%% err%n",
                    weight, total[0], stable, total[1], canary);
            if (canary > 1.0) {
                System.out.println("  analysis FAILED (error rate > 1%) -> abort, all traffic back to stable");
                return;
            }
            System.out.println(weight == 100 ? "  rollout complete: " + version + " is the new stable" : "  analysis passed -> next step");
        }
    }
}

Output

== canary order-service 2.1.0 ==
weight  10% | stable 1801 req 0.50% err | canary  199 req 0.00% err
  analysis passed -> next step
weight  25% | stable 1487 req 0.47% err | canary  513 req 0.39% err
  analysis passed -> next step
weight  50% | stable  995 req 0.40% err | canary 1005 req 0.40% err
  analysis passed -> next step
weight 100% | stable    0 req 0.00% err | canary 2000 req 0.40% err
  rollout complete: 2.1.0 is the new stable
== canary order-service 2.2.0 ==
weight  10% | stable 1801 req 0.50% err | canary  199 req 6.03% err
  analysis FAILED (error rate > 1%) -> abort, all traffic back to stable

Argo Rollout

Plain Deployments cannot split traffic by percentage. Argo Rollouts (or Flagger, or an Istio VirtualService) adds this, and can query Prometheus to decide automatically:

k8s/order-service/rollout.yaml

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: order-service
  namespace: shopease
spec:
  replicas: 10
  selector:
    matchLabels:
      app: order-service
  template:                         # same Pod template as the Deployment
    metadata:
      labels:
        app: order-service
    spec:
      containers:
        - name: order-service
          image: ghcr.io/shopease/order-service:2.1.0
  strategy:
    canary:
      steps:
        - setWeight: 10
        - pause: { duration: 10m }
        - analysis:
            templates:
              - templateName: order-error-rate     # PromQL: 5xx rate < 1%
        - setWeight: 50
        - pause: { duration: 10m }
        - setWeight: 100

Example output

$ kubectl argo rollouts get rollout order-service -n shopease
Name:            order-service
Status:          ॥ Paused
Strategy:        Canary
  Step:          2/6
  SetWeight:     10
  ActualWeight:  10
Images:          ghcr.io/shopease/order-service:2.0.4 (stable)
                 ghcr.io/shopease/order-service:2.1.0 (canary)