Pushing From PERMISSIVE Up to STRICT

Goal

You work by hand with the three scopes and the priority of mTLS policies, and leave as a plan the procedure for raising a running mesh to STRICT without cutting it off.

Why it matters

Moving to STRICT is technically changing one field, but in operations it is one of the tasks with the most frequent incidents. The reason is simple — every path that was coming in over plaintext is cut off without warning. Crons without sidecars, batch jobs outside the mesh, and custom probes do not show up well on dashboards, and then appear as an outage the moment you turn on STRICT. That is why the standard is to keep PERMISSIVE, do a full survey of the sources of plaintext traffic, and then raise from the narrow scopes first.

Another thing you must learn is the distinction of direction. PeerAuthentication handles the receiving side (inbound), and trafficPolicy.tls of a DestinationRule handles the sending side (outbound). If these two are out of step, the connection fails, and in the application log it just looks like a 503. On top of that, istioctl analyze does not catch this combination — there used to be a dedicated analyzer, but it is not in the analyzer list of the current version. So the habit of putting the two resources side by side as a pair and cross-checking them by hand becomes the safeguard itself.

In this environment no real handshake happens, so grading looks at the names, scopes, and fields of policies and the static analysis results.

Steps

Preparation before starting. Lab Pods come up fresh for each lab, so the mesh configuration from the earlier lab does not remain. If kubectl get crd peerauthentications.security.istio.io is empty, run istioctl manifest generate --set profile=minimal > /root/istio/manifest.yaml and then kubectl apply -f /root/istio/manifest.yaml twice. This manifest also creates the istio-system namespace, which must exist because step 2 puts the mesh-wide policy there. Then prepare the working namespace with kubectl create ns mesh-lab && kubectl label ns mesh-lab istio-injection=enabled.

Create /root/istio/mtls/pa-permissive.yaml — a single-document PeerAuthentication with metadata.name as default, metadata.namespace as mesh-lab, and spec.mtls.mode as PERMISSIVE, and do not put a selector (it is a whole-namespace policy). After creating it, apply it.
Apply a PeerAuthentication named default to the istio-system namespace with spec.mtls.mode: STRICT and without a selector. And in /root/istio/mtls/out/scope-note.txt, write the rule that among the three layers of mesh-wide / namespace / workload, the narrower, more specific scope takes priority.
Create the payments workload in mesh-lab — a ServiceAccount payments, a Service payments (port names http 8080 and http-metrics 9090), and a Deployment payments (Pod label app: payments, serviceAccountName: payments). Then create a PeerAuthentication legacy-metrics in mesh-lab: selector.matchLabels.app is payments, spec.mtls.mode is STRICT, and in portLevelMtls put only the single port 9090 as mode: DISABLE.
Create a DestinationRule payments-mtls in mesh-lab. spec.host is payments and spec.trafficPolicy.tls.mode is ISTIO_MUTUAL. And in /root/istio/mtls/out/direction-note.txt, write the difference that PeerAuthentication handles the receiving side (inbound) and a DestinationRule handles the sending side (client/outbound).
In /root/istio/mtls/identity.txt, write the payments workload's SPIFFE identity on exactly one line: spiffe://cluster.local/ns/mesh-lab/sa/payments. And in /root/istio/mtls/out/identity-note.txt, write the point that the mesh's identity is a service account, not an IP address.
Create a RequestAuthentication jwt-payments in mesh-lab. selector.matchLabels.app is payments, jwtRules[0].issuer is https://idp.labhub.example/, and jwtRules[0].jwksUri is https://idp.labhub.example/.well-known/jwks.json. And in /root/istio/mtls/out/jwt-note.txt, write the fact that this resource alone cannot block requests without a token and an AuthorizationPolicy is needed together.
Change the default PeerAuthentication in mesh-lab to STRICT and apply it. In that state, apply a DestinationRule with spec.trafficPolicy.tls.mode: DISABLE (for example payments-plaintext) and save the result of istioctl analyze -n mesh-lab to /root/istio/mtls/out/analyze-conflict.txt. The analyzer list of this version has no analyzer that looks at mTLS combination conflicts, so that conflict does not appear in the output — append, in one line, what conflicts with what, and leave it in the record. Then delete that DestinationRule or change it to ISTIO_MUTUAL to remove the conflict, analyze again, and save it to /root/istio/mtls/out/analyze-fixed.txt. No DestinationRule with tls.mode: DISABLE may remain in mesh-lab.
Write /root/istio/mtls/migration.md at 400 bytes or more. In the document, the discussion of PERMISSIVE and observation (metrics/monitoring) must come before (on an earlier line than) the word STRICT, and it must include staged application per namespace and a rollback (reverting) method. Finally, finish with the default PeerAuthentication in mesh-lab in the STRICT state.

Notes

Lab Pods come up fresh for each lab, so the cluster state from the earlier lab does not remain. That is why keeping mesh configuration as manifests is itself reproducibility. This is especially so for security policy — a STRICT turned on by hand vanishes with the cluster, but a policy kept in a repository is reproduced as it is on the next cluster.
In step 8, if you write STRICT first in the document title, the order check catches it. Keep the title like "mTLS migration plan" and write the body in the order stage 0 (observe, keep PERMISSIVE) → stage 1 (remove plaintext sources) → stage 2 (STRICT per namespace) → stage 3 (mesh-wide STRICT) → rollback.
In real operations, you see the plaintext ratio by the connection_security_policy label of istio_requests_total. There are no metrics in this environment, but it is right to write the basis of that judgment in the plan.
For the Deployment in step 3, it is convenient to copy /opt/lab/fixtures/istio/inject-target.yaml and add only serviceAccountName.
Common mistake 1: attaching a selector to a whole-namespace policy. The moment a selector is attached, it becomes a workload-level policy and does not apply to the other workloads.
Common mistake 2: putting a mesh-wide policy in just any namespace. It becomes mesh-wide only when it is put in the root namespace (istio-system) with the name default.

Set the namespace default policy to PERMISSIVE

Create /root/istio/mtls/pa-permissive.yaml — a single-document PeerAuthentication with metadata.name as default, metadata.namespace as mesh-lab, and spec.mtls.mode as PERMISSIVE, and do not put a selector (it is a whole-namespace policy). After creating it, apply it.

A policy that applies to a whole namespace has a fixed name and has no selector. The transition starts from a state that accepts both plaintext and ciphertext.

Raise the mesh-wide default to STRICT

Apply a PeerAuthentication named default to the istio-system namespace with spec.mtls.mode: STRICT and without a selector. And in /root/istio/mtls/out/scope-note.txt, write the rule that among the three layers of mesh-wide / namespace / workload, the narrower, more specific scope takes priority.

A mesh-wide policy goes in the root namespace. And summarize in a note which of the three scopes wins.

Open only one port of one workload as an exception

Create the payments workload in mesh-lab — a ServiceAccount payments, a Service payments (port names http 8080 and http-metrics 9090), and a Deployment payments (Pod label app: payments, serviceAccountName: payments). Then create a PeerAuthentication legacy-metrics in mesh-lab: selector.matchLabels.app is payments, spec.mtls.mode is STRICT, and in portLevelMtls put only the single port 9090 as mode: DISABLE.

Keep exceptions as narrow as possible. Select the workload with a selector, keep the default still STRICT, and specify separately only the port that is a problem.

Declare mTLS on the sending side

Create a DestinationRule payments-mtls in mesh-lab. spec.host is payments and spec.trafficPolicy.tls.mode is ISTIO_MUTUAL. And in /root/istio/mtls/out/direction-note.txt, write the difference that PeerAuthentication handles the receiving side (inbound) and a DestinationRule handles the sending side (client/outbound).

The receiving side and the sending side are handled by different resources. There is a separate mode that means using the certificate issued by the mesh instead of specifying a certificate directly.

Write down the workload's identity

In /root/istio/mtls/identity.txt, write the payments workload's SPIFFE identity on exactly one line: spiffe://cluster.local/ns/mesh-lab/sa/payments. And in /root/istio/mtls/out/identity-note.txt, write the point that the mesh's identity is a service account, not an IP address.

Three pieces go in, in order. And the object that is the basis of that identity must actually exist in the cluster.

Create an end-user JWT validation policy

Create a RequestAuthentication jwt-payments in mesh-lab. selector.matchLabels.app is payments, jwtRules[0].issuer is https://idp.labhub.example/, and jwtRules[0].jwksUri is https://idp.labhub.example/.well-known/jwks.json. And in /root/istio/mtls/out/jwt-note.txt, write the fact that this resource alone cannot block requests without a token and an AuthorizationPolicy is needed together.

To verify a signature you need both the issuer and the location of the public keys. And write in the note what this resource alone cannot block.

Create and resolve a conflict between STRICT and DISABLE

Change the default PeerAuthentication in mesh-lab to STRICT and apply it. In that state, apply a DestinationRule with spec.trafficPolicy.tls.mode: DISABLE (for example payments-plaintext) and save the result of istioctl analyze -n mesh-lab to /root/istio/mtls/out/analyze-conflict.txt. The analyzer list of this version has no analyzer that looks at mTLS combination conflicts, so that conflict does not appear in the output — append, in one line, what conflicts with what, and leave it in the record. Then delete that DestinationRule or change it to ISTIO_MUTUAL to remove the conflict, analyze again, and save it to /root/istio/mtls/out/analyze-fixed.txt. No DestinationRule with tls.mode: DISABLE may remain in mesh-lab.

If the receiving side requires encryption but the sending side insists on plaintext, the connection fails. Keep the analysis result separately before and after the fix, and do not leave any inconsistent configuration behind.

Write the staged migration plan and finish the migration

Write /root/istio/mtls/migration.md at 400 bytes or more. In the document, the discussion of PERMISSIVE and observation (metrics/monitoring) must come before (on an earlier line than) the word STRICT, and it must include staged application per namespace and a rollback (reverting) method. Finally, finish with the default PeerAuthentication in mesh-lab in the STRICT state.

The order in which words appear in the document is the order of the stages. Write observation and PERMISSIVE first and STRICT after, and include the way to revert too.