Roll Back Once Before Deleting the Old Version

Goal

Upgrade a real mesh running Istio 1.30.5 to 1.31.0 with the revision canary approach. You stand the two control planes up side by side, move just one namespace first, move the rest with a tag and then practice a rollback, and delete the old version at the end.

Why it matters

A control plane upgrade changes the configuration source of the whole mesh. With an in-place upgrade, if a problem occurs everyone experiences it at once and it is also hard to roll back. With revisions and tags, you can narrow the impact to a namespace at a time and make the rollback a single line — but only when you keep the order.

Steps

Put up the materials (web in shop and client in canary — both namespaces have istio.io/rev=prod-stable) with kubectl apply -f /opt/fixtures/istlab/upgrade-app.yaml and wait until they are ready. Then write three lines into /root/istlab-upgrade/01-before.txt: web_proxy= and client_proxy= (the tag of each Pod's istio-proxy image) and prod_stable= (the revision the tag prod-stable points to in istioctl tag list).
Save the output of istioctl x precheck into /root/istlab-upgrade/02-precheck.txt.
Install 1.31.0 as the revision 1-31-0 — istioctl install --set profile=minimal --set revision=1-31-0 --set meshConfig.accessLogFile=/dev/stdout -y. After the install, the two istiods (istiod-1-30-5 and istiod-1-31-0) must be running together, and nothing about the workloads should have changed yet.
Change the label of the canary namespace to istio.io/rev=1-31-0 and restart the client. Then write three lines into /root/istlab-upgrade/04-mixed.txt: client_proxy= and web_proxy= (each proxy image tag) and code= (the status code of http://web.shop/ from the client).
Move both the tags prod-stable and default to the revision 1-31-0 (istioctl tag set <태그> --revision 1-31-0 --overwrite, where the placeholder is the tag). Then put the canary label back to istio.io/rev=prod-stable and restart shop's web so that it receives the 1.31.0 proxy.
Before deleting, confirm that you can roll back — move the tag prod-stable to 1-30-5, restart web, and write the proxy tag into /root/istlab-upgrade/06-rollback.txt as after_rollback=, then move it to 1-31-0 again, restart web, and append after_forward=.
After confirming that every proxy is 1.31.0, delete the old revision with istioctl uninstall --revision 1-30-5 -y and wait until the istiod-1-30-5 Deployment and the injection webhook istio-sidecar-injector-1-30-5 are gone. Even after that, http://web.shop/ from canary's client must be 200.
Write five lines into /root/istlab-upgrade/08-report.md — old_revision= and new_revision= (the revision names), mixed_versions_code= (the code of step 4), rollback_version= (step 6's after_rollback), and old_injector_left= (yes if the istio-sidecar-injector-1-30-5 webhook still remains now) — and write what you learned below that in at least four lines.

Notes

This VM takes 2–4 minutes to prepare. Only revision 1-30-5 (tag prod-stable) is installed, and the 1.31.0 distribution was downloaded to /usr/local/istio-1.31.0. The istioctl in the login shell is 1.31.0.
You look at the proxy version in the VERSION column of istioctl proxy-status or in the istio-proxy image tag of the Pod. The proxy-status right after a restart may briefly show the old Pod, so it is surer to check by the image.
After changing a label, be sure to recreate the Pods with kubectl rollout restart. Injection happens when a Pod is created.
Common mistake — deleting the old revision before confirming the rollback. After that, to roll back you have to reinstall the old version.

Write down what points to what right now

Put up the materials (web in shop and client in canary — both namespaces have istio.io/rev=prod-stable) with kubectl apply -f /opt/fixtures/istlab/upgrade-app.yaml and wait until they are ready. Then write three lines into /root/istlab-upgrade/01-before.txt: web_proxy= and client_proxy= (the tag of each Pod's istio-proxy image) and prod_stable= (the revision the tag prod-stable points to in istioctl tag list).

This VM's control plane is installed only as the revision 1-30-5, and the workloads receive injection not through the revision name but through the tag prod-stable. A tag is a label that points to 'the revision we are using in production right now', so an upgrade is not fixing workload labels one by one but moving this label. This VM's istioctl is 1.31.0, but the tag list can be read regardless of the version.

Ask before upgrading — precheck

Save the output of istioctl x precheck into /root/istlab-upgrade/02-precheck.txt.

The precheck looks at whether the cluster is ready to receive the new version (the Kubernetes version, permissions, deprecated fields of the old configuration, and so on). If a warning comes up here, the order is to fix that before installing. If you run it with the 1.31.0 istioctl, it looks from the 1.31.0 standpoint.

Stand the new version next to the old one

Install 1.31.0 as the revision 1-31-0 — istioctl install --set profile=minimal --set revision=1-31-0 --set meshConfig.accessLogFile=/dev/stdout -y. After the install, the two istiods (istiod-1-30-5 and istiod-1-31-0) must be running together, and nothing about the workloads should have changed yet.

If you install with a revision, an istiod and an injection webhook (istio-sidecar-injector-1-31-0) with the revision in their names are newly created and the old ones are left as they are. No namespace points at the new revision, so at this moment there is no change to traffic — that is why the install itself is safe to do at any time. In istioctl tag list, the new revision appears as one line with no tag.

Move just one namespace to the new version

Change the label of the canary namespace to istio.io/rev=1-31-0 and restart the client. Then write three lines into /root/istlab-upgrade/04-mixed.txt: client_proxy= and web_proxy= (each proxy image tag) and code= (the status code of http://web.shop/ from the client).

Changing only the label does nothing. A sidecar is injected when the Pod is created, so you have to restart to receive the new revision's proxy. Moving just one namespace first like this is a canary — even if a problem occurs, you only have to roll back that namespace. The official docs state that data planes are currently compatible across all versions (with the caveat that this may change in the future), so mTLS must be established even when they are mixed.

Move the label — bring up the rest with the tag

Move both the tags prod-stable and default to the revision 1-31-0 (istioctl tag set <태그> --revision 1-31-0 --overwrite, where the placeholder is the tag). Then put the canary label back to istio.io/rev=prod-stable and restart shop's web so that it receives the 1.31.0 proxy.

A tag is in fact a single injection webhook (istio-revision-tag-prod-stable). If you move the tag, the istiod that webhook points at changes, and Pods newly created in namespaces that use that tag receive the new revision's proxy. If you also put back by tag the canary you had moved for the canary test, from now on you only need to manage one tag. The default tag decides the namespaces that use the istio-injection=enabled label and istioctl's default target.

Practice the rollback

Before deleting, confirm that you can roll back — move the tag prod-stable to 1-30-5, restart web, and write the proxy tag into /root/istlab-upgrade/06-rollback.txt as after_rollback=, then move it to 1-31-0 again, restart web, and append after_forward=.

While the old revision is alive, a rollback is one tag line and one restart. After you delete the old revision, you have to reinstall that version. So in production you delete the old revision only after you have lived with the new version long enough, and before that you try once whether the path back really works. You confirm that the proxy changed after the restart by the Pod's istio-proxy image.

Delete the old version after everything has moved

After confirming that every proxy is 1.31.0, delete the old revision with istioctl uninstall --revision 1-30-5 -y and wait until the istiod-1-30-5 Deployment and the injection webhook istio-sidecar-injector-1-30-5 are gone. Even after that, http://web.shop/ from canary's client must be 200.

An uninstall that specifies a revision deletes only that revision's control plane and does not touch other revisions or workloads. If Pods still using the old revision's proxy remain, those Pods lose the place to get their configuration — so before deleting, you check with istioctl proxy-status and the Pod images that nothing is left. Right after deleting, it may briefly look like 0/1, so wait until it is gone.

Leave an upgrade record

Write five lines into /root/istlab-upgrade/08-report.md — old_revision= and new_revision= (the revision names), mixed_versions_code= (the code of step 4), rollback_version= (step 6's after_rollback), and old_injector_left= (yes if the istio-sidecar-injector-1-30-5 webhook still remains now) — and write what you learned below that in at least four lines.

In the explanation lines, write in your own words 'what you gain by using a revision instead of an in-place upgrade' and 'when you delete the old revision'.