TT Lab
Get started
Learn Learning paths Courses

ICA — Istio Certified Associate

A New Control Plane Is Up, Yet No Pod Moved Over

Continue in TT Lab

Goal

You bring up two sets of the Istio control plane side by side as revisions, move workloads from the old revision (stable) to the new revision (canary) using namespace labels and the default tag, and then remove the old revision while protecting the shared resources. You also check the installation customization (IstioOperator) through the rendered result.

Why it matters

An in-place upgrade swaps the control plane all at once. If a problem occurs, the whole mesh shakes together. A canary upgrade stands up a new control plane alongside and moves over namespace by namespace, so rolling back is easy, but in exchange you must know exactly "which control plane injects into this namespace right now." There are three incidents that often occur in the field.

What is real and what is not in this lab environment

The cluster of this Pod is kwok. The API server, webhook selection, and CRDs are real, but there is no istiod process or container. So when you request Pod creation with --dry-run=server, the API server actually picks and calls the injection webhook by the namespace label, and because there is no istiod, the call fails and leaves in the error message which webhook tried to go to which istiod service. You use this as the basis for judgment. Conversely, whether a sidecar actually attaches to the new istiod cannot be seen in this environment (that is covered in the VM lab "Watch What the Mesh Actually Enforces"). Both revisions are built with the same istioctl 1.24.2.

Steps

  1. Write a custom installation configuration with an IstioOperator and render the manifest.
  2. Apply the control plane for the stable revision.
  3. Confirm that a revision installation alone does not inject istio-injection=enabled, and solve it with the default tag.
  4. Bring up the canary revision with the same customization, and confirm that installation alone does not move any namespace.
  5. Move the payments namespace to canary (the old-label trap).
  6. Inject offline with the canary injection configuration and see which control plane the sidecar is made to attach to.
  7. Move the default tag to canary to move the legacy namespace all at once.
  8. Leave the shared resources and remove only stable.

Notes

Extracting in advance what will be installed, before installing

In /root/ica-upgrade/stable.yaml, write an IstioOperator: profile: minimal, revision: stable, meshConfig.accessLogFile: /dev/stdout, and istiod (pilot) container requests of cpu: 250m and memory: 512Mi. Then save the result rendered with istioctl manifest generate -f to /root/ica-upgrade/stable-manifest.yaml. Do not apply it to the cluster yet.

The IstioOperator's spec.revision becomes a suffix attached after the names of the control plane objects. Put the mesh-wide configuration in spec.meshConfig and a component's Kubernetes settings under spec.components.<구성요소>.k8s (the placeholder is the component). In the rendered result, find the Deployment name and requests and the ConfigMap's mesh configuration with yq, and check that the values you wanted went in.

Bringing up the stable revision control plane

Create the namespace istio-system and apply stable-manifest.yaml to the cluster. The Istio CRDs, the istiod-stable Deployment and Service, and the istio-sidecar-injector-stable webhook configuration must appear.

This cluster is kwok, so the istiod Pod appears Running but there is no actual process. Even so, the API server, webhook configuration, and CRDs are real. After applying, read from kubectl get mutatingwebhookconfiguration -o yaml which service the webhook calls (clientConfig.service) and which namespace labels it selects (namespaceSelector) — it is the key to the next step.

It is istio-injection=enabled, yet no sidecar is attached

Attach the label istio.io/rev=stable to the namespace shop and the label istio-injection=enabled to the namespace legacy. Check which injection webhook Pod creation goes to with kubectl -n <ns> run probe --image=registry.example.invalid/app:1 --restart=Never --dry-run=server and save the output (including the error) — the legacy result goes in probe-legacy-before.txt. Next, save the output of istioctl tag generate default --revision stable as tag-default.yaml and apply it, then save legacy again in probe-legacy-after.txt and shop in probe-shop.txt (all under /root/ica-upgrade/).

The injection webhook of an installation carrying a revision name selects only the istio.io/rev=<리비전> label (the placeholder is the revision). The old-style istio-injection=enabled means the default revision, and if you install only a revision, no webhook takes on that role. The default tag fills that place. A dry-run=server request also goes through the mutating webhook, so on this cluster with no istiod, if the webhook was called, the failure message shows the webhook name and the istiod service address, and if it was not called, it simply says created. A webhook call has a 10-second limit, so wait a little.

We brought up the new control plane, but nobody moved over

In /root/ica-upgrade/canary.yaml, write an IstioOperator with revision: canary and the same customization as stable (accessLogFile and requests), and apply the rendered canary-manifest.yaml. Create the namespace payments (label istio-injection=enabled), and as a sample to see whether existing configuration survives during the upgrade, create a VirtualService payments/api (hosts [api], destination host api) and write its uid in /root/ica-upgrade/sentinel-uid.txt. Finally, dry-run shop again and save it to probe-shop-after-canary.txt — merely installing a new revision must not move shop.

The two revisions exist side by side because their name suffixes differ. Which side a namespace goes to is decided by labels and tags, not by installation. Also check by comparing the two manifests that the CRDs are a shared resource of which there is only one on the cluster regardless of revision. Right now the default tag's validation webhook is called on every Istio resource write (ignored on failure), so creating a VirtualService takes about 10 seconds.

I changed the label, but it still goes to the old control plane

Move the namespace payments to canary. In the end, payments must have istio.io/rev=canary and must not have the istio-injection label, and the dry-run result must go to canary's istiod. Save the dry-run output after moving to /root/ica-upgrade/probe-payments.txt.

Try a dry-run after adding only istio.io/rev=canary. When both labels are present, which webhook wins — what the condition on istio-injection is in the revision webhook's namespaceSelector — is the answer. The official canary upgrade documentation also tells you to remove the old label for the same reason. On a real cluster, you would have to restart the Pods after this for new sidecars to be injected.

Where is it written that you must restart for it to move over

Write a Deployment api (namespace payments, 1 replica, container api with image nginx:1.27) in /root/ica-upgrade/api.yaml, and save the result of istioctl kube-inject with the canary revision's injection configuration to /root/ica-upgrade/api-injected.yaml. Take the injection configuration from the cluster's ConfigMaps istio-sidecar-injector-canary (keys config and values) and istio-canary (key mesh) and pass it as files. Do not apply the result to the cluster.

By default kube-inject attaches to istiod to receive the configuration, but this cluster has no istiod process. Instead, if you give all three files, --injectConfigFile, --valuesFile, and --meshConfigFile, it renders offline. In the result, find which address (discoveryAddress) of the control plane istio-proxy was made to attach to. That value is baked in when the Pod is created, so a Pod that is already running stays attached to the old istiod even if you change the namespace label.

Moving even the teams that use the old label in one go

Change the default tag to point to canary (save it as /root/ica-upgrade/tag-default-canary.yaml and apply it). Without touching the label of the namespace legacy, the dry-run of legacy must go to canary's istiod. Save that output to /root/ica-upgrade/probe-legacy-canary.txt.

A tag is a label between a namespace label and a revision. Instead of changing labels in dozens of namespaces, if you change only the revision the tag points to, everywhere that uses that label moves together. Read the error message to see what istioctl requires when recreating a tag that already exists.

When we deleted the old control plane, even the configuration was nearly lost

Remove stable. First move the namespace still using stable (shop) to canary, and delete the objects that are only in the stable manifest. You must not delete the objects that both revisions share (the Istio CRDs and the ServiceAccount istio-reader-service-account) — if you delete a CRD, all resources of that kind (including the VirtualService payments/api) vanish with it. Leave the list of deleted objects (one per line, in the format 종류/이름, meaning kind and name) in /root/ica-upgrade/retired.txt.

If you extract a list of 종류/이름 (kind and name) from the two manifests and compare them, what is only in stable and what is shared separates (extract with yq and compare with comm). This lab applied with kubectl, so you revert using the same manifest. The official documentation's istioctl uninstall --revision is a command for clusters installed with istioctl, and when measured on this cluster, it did not find anything to delete. After deleting, check that shop's dry-run goes to canary and that the sample VirtualService's uid is unchanged.