We Changed the Root and Only Half Could Talk

Goal

Replace istiod's self-signed root with a root and intermediate CA we made (a plug-in CA), and confirm directly from the certificates that the workloads between which only one side received a new certificate are cut off at the moment of the change, and the chain, identity, and lifetimes after everything has rotated.

Why it matters

A mesh's mTLS stands on the premise that all workloads trust the same root. Moving that root to the organization's PKI is essential for security, but if you do it without knowing the order, half of the connections between services get cut. Once you have seen the break directly, you can explain why 'a period of trusting both roots' is needed.

Steps

Put up the materials with kubectl apply -f /opt/fixtures/istlab/pluginca-app.yaml and wait until they are ready. Then write three lines into /root/istlab-ca/01-before.txt — root_subject= (the subject of the root in the ConfigMap istio-ca-root-cert of the namespace bank), root_sha256= (that root's SHA-256 fingerprint, the value part of openssl x509 -fingerprint -sha256), and leaf_issuer= (the issuer of the workload certificate the client sidecar received).
In /root/istlab-ca/certs, make a root (make -f … root-ca) and an intermediate CA for the cluster cluster1 (make -f … cluster1-cacerts) with the distribution's /usr/local/istio-1.31.0/tools/certs/Makefile.selfsigned.mk. /root/istlab-ca/certs/root-cert.pem and, under /root/istlab-ca/certs/cluster1/, ca-cert.pem, ca-key.pem, root-cert.pem, and cert-chain.pem must be created.
Create the Secret cacerts in istio-system — with the four files of /root/istlab-ca/certs/cluster1/ (ca-cert.pem, ca-key.pem, root-cert.pem, and cert-chain.pem) as keys of the same names. Then restart istiod and wait until it is ready. You do not restart the workloads yet.
Restart only web. Then write three lines into /root/istlab-ca/04-mixed.txt — code= (the status code of http://web/ from the client), client_root= (the ROOTCA subject of the client sidecar), and web_leaf_issuer= (the issuer of the web sidecar's workload certificate).
Restart the client as well so that client to web is 200 again. Then save the client sidecar's workload certificate chain (the certificateChain of default) as PEM in /root/istlab-ca/05-chain.pem.
From the first certificate of /root/istlab-ca/05-chain.pem (the workload certificate), write the SAN's URI into /root/istlab-ca/06-identity.txt as spiffe= and that URI's trust domain (the first part after spiffe://) as trust_domain=.
Write three lines into /root/istlab-ca/07-lifetimes.txt — leaf_hours= (the workload certificate's notAfter minus notBefore in hours, decimals dropped), intermediate_days= (the intermediate CA's validity period in days), and root_days= (the root's validity period in days).
Write five lines into /root/istlab-ca/08-report.md — old_root_subject= (step 1), new_root_subject= (the subject of /root/istlab-ca/certs/root-cert.pem), mixed_code= (step 4), after_restart_code= (the status code of client to web now), and leaf_hours= (step 7) — and write what you learned below that in at least four lines.

Notes

This VM takes 2–4 minutes to prepare. The minimal profile (istiod only) is installed, and the Makefile for certificates is at /usr/local/istio-1.31.0/tools/certs/Makefile.selfsigned.mk.
You look at the certificates the sidecar received in istioctl proxy-config secret deploy/client -n bank -o json as default (the workload certificate chain) and ROOTCA (the root it trusts), and the values come in base64.
You restart with kubectl -n bank rollout restart deploy/<이름> (the placeholder is the name). In step 4 you restart web only.
Common mistake — creating cacerts and not restarting istiod. istiod looks at cacerts only when it starts.

Who made the current root

Put up the materials with kubectl apply -f /opt/fixtures/istlab/pluginca-app.yaml and wait until they are ready. Then write three lines into /root/istlab-ca/01-before.txt — root_subject= (the subject of the root in the ConfigMap istio-ca-root-cert of the namespace bank), root_sha256= (that root's SHA-256 fingerprint, the value part of openssl x509 -fingerprint -sha256), and leaf_issuer= (the issuer of the workload certificate the client sidecar received).

If there is no cacerts, istiod makes a root itself and puts it in istio-system/istio-ca-secret, and distributes that root as the istio-ca-root-cert ConfigMap of every namespace. The sidecar asks istiod for and receives the workload certificate, and you can see it in default (the workload certificate chain) and ROOTCA (the root it trusts) of istioctl proxy-config secret deploy/client -n bank -o json. The values come in base64.

Make a root and a cluster intermediate CA

In /root/istlab-ca/certs, make a root (make -f … root-ca) and an intermediate CA for the cluster cluster1 (make -f … cluster1-cacerts) with the distribution's /usr/local/istio-1.31.0/tools/certs/Makefile.selfsigned.mk. /root/istlab-ca/certs/root-cert.pem and, under /root/istlab-ca/certs/cluster1/, ca-cert.pem, ca-key.pem, root-cert.pem, and cert-chain.pem must be created.

The structure the Istio docs recommend is 'keep the root offline and give istiod, per cluster, an intermediate CA that root signed'. Then even if one cluster's key leaks, you do not have to replace the root, and several clusters trust the same root so they can trust each other's workloads. The docs state that this Makefile is for demonstration and that in production you should use a CA such as Vault. After making them, check the chain with openssl verify -CAfile root-cert.pem cluster1/ca-cert.pem.

Plug in cacerts and bring istiod up again

Create the Secret cacerts in istio-system — with the four files of /root/istlab-ca/certs/cluster1/ (ca-cert.pem, ca-key.pem, root-cert.pem, and cert-chain.pem) as keys of the same names. Then restart istiod and wait until it is ready. You do not restart the workloads yet.

istiod checks when it starts whether cacerts exists, and if it does, it signs with that intermediate CA instead of the root it made itself. A running istiod does not notice even if the Secret appears, so you restart it. A line saying it read the root from cacerts is left in the newly started istiod's log, and each namespace's istio-ca-root-cert changes to the new root — but what happens to the root the already running sidecars trust, you see in step 4.

If only one side receives a new certificate

Restart only web. Then write three lines into /root/istlab-ca/04-mixed.txt — code= (the status code of http://web/ from the client), client_root= (the ROOTCA subject of the client sidecar), and web_leaf_issuer= (the issuer of the web sidecar's workload certificate).

The new istiod gives the newly started web a certificate signed by the new intermediate CA. But the sidecar of the client that was not restarted still trusts only the old root. mTLS has both sides verify each other's certificate, so if even one side does not know the other's root, the connection is not established. This is why, when you change the root in production, you set up a period of trusting both the old root and the new root.

Make everyone receive a new certificate

Restart the client as well so that client to web is 200 again. Then save the client sidecar's workload certificate chain (the certificateChain of default) as PEM in /root/istlab-ca/05-chain.pem.

The restarted client trusts the new root and receives a certificate signed by the new intermediate CA. The saved chain is in the order workload certificate, then intermediate CA, then root, so you can check that it leads up to our root with openssl verify -CAfile /root/istlab-ca/certs/root-cert.pem -untrusted /root/istlab-ca/05-chain.pem /root/istlab-ca/05-chain.pem.

The identity the certificate speaks

From the first certificate of /root/istlab-ca/05-chain.pem (the workload certificate), write the SAN's URI into /root/istlab-ca/06-identity.txt as spiffe= and that URI's trust domain (the first part after spiffe://) as trust_domain=.

Istio's workload certificate has an empty subject, and the identity is carried in a single SAN URI (spiffe://<신뢰 도메인>/ns/<네임스페이스>/sa/<서비스어카운트>, where the placeholders are the trust domain, the namespace, and the service account). An authorization policy's principals is exactly this value (the part without the scheme). Even if you change the CA, you do not need to fix the policy as long as the trust domain is the same. You look with openssl x509 -noout -ext subjectAltName.

Who lives how long

Write three lines into /root/istlab-ca/07-lifetimes.txt — leaf_hours= (the workload certificate's notAfter minus notBefore in hours, decimals dropped), intermediate_days= (the intermediate CA's validity period in days), and root_days= (the root's validity period in days).

The workload certificate is given short by istiod (24 hours by default), and the sidecar swaps it on its own before it expires. So the damage period is short even if it leaks, and workloads that were not restarted after changing the CA will soon receive new certificates too — though a break like step 4 can continue until then. The intermediate CA and the root are managed by people, so they live long. You read the dates with openssl x509 -noout -startdate -enddate and convert to seconds with date -d and subtract.

Write down the order for changing a root

Write five lines into /root/istlab-ca/08-report.md — old_root_subject= (step 1), new_root_subject= (the subject of /root/istlab-ca/certs/root-cert.pem), mixed_code= (step 4), after_restart_code= (the status code of client to web now), and leaf_hours= (step 7) — and write what you learned below that in at least four lines.

In the explanation lines, write in your own words 'what order is needed to change the root on a mesh that is already in operation (a period of trusting both the old root and the new root)'. This lab changed it without that period and deliberately saw the break.