Adding a policy exposed the payment API

Goal

On real Istio sidecars, you verify the combination of workload identity and JWT principal, and fix the incident in which the allowed scope became too broad.

Why it matters

You cannot confirm security from a file name or from kubectl apply succeeding. With real requests in which you change the source, token, method, path, and port, you check together that legitimate access is allowed and bypasses are denied. The four comparison servers stay to the end and are checked again during the overall grading.

Environment and safety scope

Change only the ica-policy namespace inside this VM. It is neither a production cluster nor an external IdP. k3s v1.35.8+k3s1 and Istio 1.31.0 are prepared, and orders, stranger, and the real containers of the four servers are running. The public key uses prepared material, and you do not put in real user tokens. union is an isolated target that is deliberately allowed wide open to reproduce the incident. When the session ends, the VM and working files are reclaimed.

All policies use security.istio.io/v1, and the request helper is python3 /opt/fixtures/ica-policy/runtime.py observe . The helper does not change policies and prints the actual HTTP code and body. Right after applying a policy, wait for propagation and observe again. Grading passes only if the same result comes out twice in a row.

Steps

List the Pods in the ica-policy namespace, pick out only the apiVersion and kind and each item's metadata.name and metadata.uid, and save them to /root/ica-policy/inventory.json. The full output that includes sidecar details can exceed the input limit of 128KiB. The six of baseline, union, intersection, scoped, orders, and stranger must actually be Ready and have sidecars. If you recreated a Pod, record the UID again too.
In /root/ica-policy/baseline.yaml, write and apply an AuthorizationPolicy allow-baseline in ica-policy. app=baseline, action=ALLOW, a single rule whose source.principals is just cluster.local/ns/ica-policy/sa/orders, and operation with one each of POST and /v1/charges. Do not put a JWT condition. With the baseline observation, check that orders' normal request and the no-token request are 200, and stranger's valid token is 403.
Reproduce an intentionally wrong configuration only on the isolated union target. In /root/ica-policy/union.yaml, write and apply an AuthorizationPolicy jwt-union in ica-policy. app=union, action=ALLOW, and in one from of one rule put only requestPrincipals=[https://issuer.example.invalid/*], with no to. Preserve the existing allow-union. In the union observation, check that even orders without a token, stranger with a valid token, and a valid orders GET /admin/test are 200. Do not apply this to production.
In /root/ica-policy/intersection.yaml, write and apply a policy allow-intersection in ica-policy. app=intersection, action=ALLOW, and in the same source of a single rule, put together principals=[cluster.local/ns/ica-policy/sa/orders] and requestPrincipals=[https://issuer.example.invalid/*]. The operation of the same rule allows only POST and /v1/charges. Replace the existing allow-intersection with this content but do not add another ALLOW. Only a valid orders must be 200, and no token, stranger, and other paths must be 403.
In /root/ica-policy/scoped.yaml, write and apply a policy jwt-scoped in ica-policy. app=scoped, action=DENY, and in a single rule put together from.source.notRequestPrincipals=[*] and to.operation.ports=["8080"]. Preserve the existing allow-scoped as is, and do not add extra conditions, rules, or a dry-run. Only a valid orders payment call must be 200, and no token, stranger, and other paths must be 403.
The initial JWT configuration of intersection allows both the payments-api and other-api audiences. After confirming this with the audience observation, write and apply a RequestAuthentication jwt-intersection in ica-policy in /root/ica-policy/authentication.yaml. app=intersection, one jwtRule with issuer=https://issuer.example.invalid, audiences=[payments-api], and jwks being the actual public key JSON string from /opt/fixtures/ica-policy/public-jwks.json. Do not put a jwksUri or an additional issuer. Check, with the body, that a valid token is 200, a different audience is 403, and a different issuer, expiry, and malformed format are 401.
Save the JSON output of the ports observation command to /root/ica-policy/ports.json. For orders calling POST /v1/charges on scoped without a token, the 8080 response must be 403/RBAC denial and the 8081 response must be 200/synthetic-order. Do not change the observation result to a success value by hand. Grading sends the current requests again and cross-checks them.
In /root/ica-policy/incident.json, write allow_composition (OR or AND between policies), source_fields (OR or AND between fields of the same source), jwt_missing_fix (the action used in scoped), deny_ports (the array of protected string ports), audience_error (the layer that rejected the JWT audience: jwt_authn or rbac), and principal_error (the layer that denied workload identity authorization: jwt_authn or rbac). Preserve the valid intersection and the scoped port boundary.

Notes

Both the saved YAML and the actually applied policies must be correct. Cross-check with kubectl -n ica-policy get authorizationpolicy,requestauthentication -o yaml.
Grading does not fix student files. Even if you write only success codes in a file, it fails if the actual requests differ.
8080 and 8081 are HTTP in this lab. Do not extend this result to verification of all TCP or an ambient mesh.

Recording the Pod identities of the current mesh

List the Pods in the ica-policy namespace, pick out only the apiVersion and kind and each item's metadata.name and metadata.uid, and save them to /root/ica-policy/inventory.json. The full output that includes sidecar details can exceed the input limit of 128KiB. The six of baseline, union, intersection, scoped, orders, and stranger must actually be Ready and have sidecars. If you recreated a Pod, record the UID again too.

Having the same name does not make it the same Pod. Look separately at the UID, Ready, and the actual istio-proxy container.

A baseline where a legitimate workload without a token gets through

In /root/ica-policy/baseline.yaml, write and apply an AuthorizationPolicy allow-baseline in ica-policy. app=baseline, action=ALLOW, a single rule whose source.principals is just cluster.local/ns/ica-policy/sa/orders, and operation with one each of POST and /v1/charges. Do not put a JWT condition. With the baseline observation, check that orders' normal request and the no-token request are 200, and stranger's valid token is 403.

Whether there is a valid JWT and which workload the request comes from are different axes. The baseline restricts only the source, method, and path.

Attaching a JWT allow separately to reproduce the incident

Reproduce an intentionally wrong configuration only on the isolated union target. In /root/ica-policy/union.yaml, write and apply an AuthorizationPolicy jwt-union in ica-policy. app=union, action=ALLOW, and in one from of one rule put only requestPrincipals=[https://issuer.example.invalid/*], with no to. Preserve the existing allow-union. In the union observation, check that even orders without a token, stranger with a valid token, and a valid orders GET /admin/test are 200. Do not apply this to production.

A new allow is not a filter that makes the existing allow stricter. Explain separately which condition lets each request through.

Combining two identities in the same source

In /root/ica-policy/intersection.yaml, write and apply a policy allow-intersection in ica-policy. app=intersection, action=ALLOW, and in the same source of a single rule, put together principals=[cluster.local/ns/ica-policy/sa/orders] and requestPrincipals=[https://issuer.example.invalid/*]. The operation of the same rule allows only POST and /v1/charges. Replace the existing allow-intersection with this content but do not add another ALLOW. Only a valid orders must be 200, and no token, stranger, and other paths must be 403.

Putting the two identities in different items of from is different from putting them in the same source.

Limiting the no-JWT DENY to the HTTP port

In /root/ica-policy/scoped.yaml, write and apply a policy jwt-scoped in ica-policy. app=scoped, action=DENY, and in a single rule put together from.source.notRequestPrincipals=[*] and to.operation.ports=["8080"]. Preserve the existing allow-scoped as is, and do not add extra conditions, rules, or a dry-run. Only a valid orders payment call must be 200, and no token, stranger, and other paths must be 403.

If only the DENY remains, the least privilege for requests that do not meet the deny condition may disappear. Look at the existing ALLOW and the blocking scope together.

Preventing reuse of a token meant for another API

The initial JWT configuration of intersection allows both the payments-api and other-api audiences. After confirming this with the audience observation, write and apply a RequestAuthentication jwt-intersection in ica-policy in /root/ica-policy/authentication.yaml. app=intersection, one jwtRule with issuer=https://issuer.example.invalid, audiences=[payments-api], and jwks being the actual public key JSON string from /opt/fixtures/ica-policy/public-jwks.json. Do not put a jwksUri or an additional issuer. Check, with the body, that a valid token is 200, a different audience is 403, and a different issuer, expiry, and malformed format are 401.

Do not create a new key. Insert the prepared public key as a string and narrow only the audience allow list to the required scope.

Telling the protected port from the port left behind

Save the JSON output of the ports observation command to /root/ica-policy/ports.json. For orders calling POST /v1/charges on scoped without a token, the 8080 response must be 403/RBAC denial and the 8081 response must be 200/synthetic-order. Do not change the observation result to a success value by hand. Grading sends the current requests again and cross-checks them.

Do not confuse port and targetPort; check the string ports list of the applied policy. This step is not a task of protecting even 8081.

Reporting the cause of the policy-composition incident

In /root/ica-policy/incident.json, write allow_composition (OR or AND between policies), source_fields (OR or AND between fields of the same source), jwt_missing_fix (the action used in scoped), deny_ports (the array of protected string ports), audience_error (the layer that rejected the JWT audience: jwt_authn or rbac), and principal_error (the layer that denied workload identity authorization: jwt_authn or rbac). Preserve the valid intersection and the scoped port boundary.

Do not treat every 403 as the same error. Report by linking the response body to which policy rejected that request.