Running a Canary With Weights, Headers and Mirroring

Goal

You become able to choose among the three splitting methods of weight, header, and mirroring to suit the situation, and to create a canary plan that states the conditions for rolling back.

Why it matters

The essence of a canary is not "give a little" but "decide first when to stop." Anyone can raise a weight, but the judgment of rolling back upon seeing bad metrics wavers if left to a person. So you write a criterion for each stage and make that criterion the input to automation. Another important thing is the separation of the number of Pods from the traffic ratio. In a mesh, you can bring up all of the new version and still keep traffic at 0%, and if a problem shows up, you can return only the weight to 0 without a deployment.

This lab mixes two kinds of grading. Steps 1, 2, 4, and 5 look at saved files, and steps 6, 7, and 8 look at the state actually applied to the cluster. So you must keep a separate file for each step, and if you keep overwriting one, the earlier steps fail again. In each vs-*.yaml file, put only one VirtualService document (if you put several documents in one file, grading cannot identify the first one).

Steps

Preparation before starting. Lab Pods come up fresh for each lab, so the mesh configuration from the earlier lab does not remain. If kubectl get crd virtualservices.networking.istio.io is empty, run istioctl manifest generate --set profile=minimal > /root/istio/manifest.yaml and then kubectl apply -f /root/istio/manifest.yaml twice, and create the namespace with kubectl create ns mesh-lab && kubectl label ns mesh-lab istio-injection=enabled. Step 6 looks up live Pods by label, so you must wait until the fixture workload's Pods are Running.

First apply /opt/lab/fixtures/istio/workload-v1v2.yaml to mesh-lab, and apply a DestinationRule reviews that has v1 (labels: {version: v1}) and v2 (labels: {version: v2}) in spec.subsets. Then create /root/istio/canary/vs-baseline.yaml — one VirtualService with exactly two items in spec.http[0].route, where the weight of subset: v1 is 100 and the weight of subset: v2 is 0.
Create /root/istio/canary/vs-90-10.yaml. v1 is weight: 90 and v2 is weight: 10 (the sum is 100). Run kubectl apply on this file and save its output to /root/istio/canary/out/applied-10.txt (one of configured/created/unchanged must be visible).
Create a file whose weight sum is not 100 (for example v1 90 / v2 20 in /root/istio/canary/vs-bad.yaml), check it with istioctl validate -f, and save its rejection message, including standard error, to /root/istio/canary/out/weight-error.txt. And in /root/istio/canary/out/weight-note.txt, write the rule that the weight sum must be 100.
Create /root/istio/canary/vs-header.yaml. Put at the front a route that selects requests whose header x-canary is exactly true, with destination subset: v2 and weight: 100. After it, put one more default route with no conditions (the conditional route must not be last).
Create /root/istio/canary/vs-mirror.yaml. spec.http[0].route is weight: 100 on subset: v1, and in the same item put mirror (host: reviews, subset: v2) and mirrorPercentage.value: 10. And in /root/istio/canary/out/mirror-note.txt, write that the response to a mirrored request is thrown away and the risk of side effects from write operations.
First reapply vs-90-10.yaml so that it references both v1 and v2. Then change the DestinationRule reviews subset name v2 temporarily to v2-canary and apply it, and save the result of istioctl analyze -n mesh-lab to /root/istio/canary/out/analyze-mismatch.txt. Change the name back to v2 (labels version: v2), apply, analyze again, and save it to /root/istio/canary/out/analyze-clean.txt (there must be no Error [ in this one).
Add trafficPolicy.loadBalancer.consistentHash.httpHeaderName: x-user-id to the DestinationRule reviews and apply it. And in /root/istio/canary/out/sticky-note.txt (at least 50 bytes), write why it is a problem if the same user goes back and forth between versions.
Create /root/istio/canary/rollout.yaml. At the top level there must be a stages array (5 or more stages) and a rollback key. Each stage must have both weight (the v2 weight) and criteria (the criterion for moving to the next stage), and weight must start at 0 and end at 100 and must not decrease in the middle. Finally apply the real VirtualService reviews as v1 50 / v2 50 (based on the last item of spec.http). Save the result of istioctl analyze -n mesh-lab to /root/istio/canary/out/final-analyze.txt, and there must be no errors.

Notes

Lab Pods come up fresh for each lab, so the cluster state from the earlier lab does not remain. That is why keeping mesh configuration as manifests is itself reproducibility. The same is why this lab asks you to keep a separate file for each step — the cluster may vanish but files remain.
First check with kubectl get pods -n mesh-lab -l app=reviews --show-labels that the Pod's version label is actually attached. The grading of step 6 finds Pods by this label.
The istioctl validate in step 3 ends in failure, so save it like istioctl validate -f 파일 > 출력 2>&1 || true (the placeholders are the file and the output file).
In step 6, if you change the subset name, the v2 that the VirtualService references disappears and the analysis catches it immediately. It is a surer experiment than changing only the label.
The rollout.yaml of step 8 is not an Istio resource but a plan file you design. The format is free, but use the key names stages, weight, criteria, and rollback as they are.
Common mistake 1: keeping rewriting one file. Only when each step has a different file does the grading of earlier steps hold.
Common mistake 2: omitting v1's weight: 100 in step 5. If there is a single destination it works even when omitted, but it is safer to state explicitly in the manifest that mirroring does not change the real response.

Create the 100/0 starting-point manifest

First apply /opt/lab/fixtures/istio/workload-v1v2.yaml to mesh-lab, and apply a DestinationRule reviews that has v1 (labels: {version: v1}) and v2 (labels: {version: v2}) in spec.subsets. Then create /root/istio/canary/vs-baseline.yaml — one VirtualService with exactly two items in spec.http[0].route, where the weight of subset: v1 is 100 and the weight of subset: v2 is 0.

It is a state in which the new version is deployed but given no traffic. If you write both subsets, from the next step on you only need to change the numbers.

Move 10% to the new version

Create /root/istio/canary/vs-90-10.yaml. v1 is weight: 90 and v2 is weight: 10 (the sum is 100). Run kubectl apply on this file and save its output to /root/istio/canary/out/applied-10.txt (one of configured/created/unchanged must be visible).

Do not overwrite the earlier step's file; keep a new file. The apply output must also be saved as evidence.

See what happens when the weight sum is wrong

Create a file whose weight sum is not 100 (for example v1 90 / v2 20 in /root/istio/canary/vs-bad.yaml), check it with istioctl validate -f, and save its rejection message, including standard error, to /root/istio/canary/out/weight-error.txt. And in /root/istio/canary/out/weight-note.txt, write the rule that the weight sum must be 100.

Deliberately create a wrong file and ask the validation tool. The rejection message itself is the output of this step. The command ends in failure, so you must save standard error too.

What gets caught and what does not is the key of this step. Even if the sum is 110, istioctl currently lets it pass — as Envoy changed to normalize by total_weight, the sum-100 rule dropped out of validation. On the other hand, a sum of 0 is rejected. Try both and leave the results together. What you learn here is that a validation tool does not catch every mistake.

Send only internal testers to the new version

Create /root/istio/canary/vs-header.yaml. Put at the front a route that selects requests whose header x-canary is exactly true, with destination subset: v2 and weight: 100. After it, put one more default route with no conditions (the conditional route must not be last).

The designated people must see the new version 100%. The conditional route must be above the default path, and the default path goes at the very end with no conditions.

Set up mirroring that throws away the response

Create /root/istio/canary/vs-mirror.yaml. spec.http[0].route is weight: 100 on subset: v1, and in the same item put mirror (host: reviews, subset: v2) and mirrorPercentage.value: 10. And in /root/istio/canary/out/mirror-note.txt, write that the response to a mirrored request is thrown away and the risk of side effects from write operations.

Mirroring does not change the response that goes to the user. The existing version handles 100% of the real traffic, and only a copy goes to the new version. Do not forget to specify the percentage.

Create and fix a mismatch between subset and Pod labels

First reapply vs-90-10.yaml so that it references both v1 and v2. Then change the DestinationRule reviews subset name v2 temporarily to v2-canary and apply it, and save the result of istioctl analyze -n mesh-lab to /root/istio/canary/out/analyze-mismatch.txt. Change the name back to v2 (labels version: v2), apply, analyze again, and save it to /root/istio/canary/out/analyze-clean.txt (there must be no Error [ in this one).

If a subset the route references disappears from the definition, static analysis catches it. Keep the analysis result for the mismatched state and for the fixed state separately.

Pin the same user to one version

Add trafficPolicy.loadBalancer.consistentHash.httpHeaderName: x-user-id to the DestinationRule reviews and apply it. And in /root/istio/canary/out/sticky-note.txt (at least 50 bytes), write why it is a problem if the same user goes back and forth between versions.

Weighted splitting draws again for every request. Change the load balancer to hash-based and decide which header to use as the key.

Make a stage-by-stage canary plan and advance to the middle

Create /root/istio/canary/rollout.yaml. At the top level there must be a stages array (5 or more stages) and a rollback key. Each stage must have both weight (the v2 weight) and criteria (the criterion for moving to the next stage), and weight must start at 0 and end at 100 and must not decrease in the middle. Finally apply the real VirtualService reviews as v1 50 / v2 50 (based on the last item of spec.http). Save the result of istioctl analyze -n mesh-lab to /root/istio/canary/out/final-analyze.txt, and there must be no errors.

The plan needs a weight and a criterion for each stage, and the whole needs a way to roll back. And advance the real mesh to the middle stage too.