Istio Deep Dive — Why It Flows That Way
Move two policies into RBAC filters and verify the evaluation order
Goal
Write two AuthorizationPolicies, DENY and ALLOW, predict the results first by the evaluation order, then translate the two policies into two Envoy RBAC filters, start them and compare with your prediction. You confirm the rules one at a time by swapping the order, removing ALLOW, and using dry-run.
Why it matters
Most authorization incidents come from misreading the evaluation order — adding one ALLOW blocks everything, or a DENY that wrote only a path cuts a TCP service. If you know how policies are translated into a filter chain, you can look at one 403 and pin down which line of which policy blocked it.
Steps
- Write an AuthorizationPolicy
deny-adminto/root/ist2-authz/deny.yaml(namespacedefault,selector.matchLabels.app: payments,action: DENY, one rule —to.operation.paths: ["/admin*"]) andallow-getto/root/ist2-authz/allow.yaml(the same namespace and selector,action: ALLOW,to.operation.methods: ["GET"]). In/root/ist2-authz, put the output and exit code ofistioctl validate -f deny.yaml -f allow.yamlin/root/ist2-authz/01-validate.txt(withrc=on the last line). - With both policies applied to
payments, predict by the evaluation order alone the verdicts for four requests —GET /,GET /admin/x,POST /andPOST /admin— and write them to/root/ist2-authz/02-predict.txtas four lines of the form<메서드> <경로>=allow|deny(the placeholders are the method and the path; for exampleHEAD /=deny). You do not start anything yet. - Write an Envoy configuration to
/root/ist2-authz/authz.yaml— admin port9987; the listenervirtualInboundis127.0.0.1:10087, the HCM'sstat_prefixisinbound_0.0.0.0_8110, and every path goes to the clusterinbound|8110||(127.0.0.1:8110).http_filtershas exactly three, in this order — (1)istio_authz_deny: RBAC,rules.action: DENY, policy namens[default]-policy[deny-admin]-rule[0], permission is theurl_pathprefix/admin; (2)envoy.filters.http.rbac: RBAC,rules.action: ALLOW, policy namens[default]-policy[allow-get]-rule[0], permission is the header:methodexactlyGET; both policies haveprincipals: [{any: true}]; (3) the router. Then readhttp_filterswithyqand write one line per filter to/root/ist2-authz/03-chain.txtas<이름> <rules.action>(the placeholder is the name;-for the router). - Start an upstream with
python3 /opt/lab/envoy/upstream.py 8110 okand start Envoy with/root/ist2-authz/authz.yaml. Send the four requests from step 2 tolocalhost:10087and write four lines of<메서드> <경로>=<HTTP 코드>(the placeholders are the method, the path and the HTTP code) to/root/ist2-authz/04-result.txt, and on the fifth line,body_403=, write the first line of the body thatGET /admin/xreceived. - Copy
/root/ist2-authz/authz.yamlto/root/ist2-authz/authz-swapped.yaml, then change only the order of the two RBAC filters (the ALLOW filter in front, the DENY filter behind, the router at the very end, everything else as it is). Start Envoy again with this file, send the same four requests, and write four lines (<메서드> <경로>=<코드>, where the placeholders are the method, the path and the code) andsame_as_04=(yesif all four codes are the same as in step 4, otherwiseno) to/root/ist2-authz/05-swapped.txt. - Make
/root/ist2-authz/authz-denyonly.yamlfrom/root/ist2-authz/authz.yamlwith only the ALLOW filter (envoy.filters.http.rbac) removed, and start Envoy again. SendPOST /andGET /admin/xand write two lines (<메서드> <경로>=<코드>, where the placeholders are the method, the path and the code) to/root/ist2-authz/06-denyonly.txt, and on the third line,evaluation_rule=, write as a number which of the five rows of the evaluation order letPOST /through. - Based on
/root/ist2-authz/authz.yaml, make/root/ist2-authz/authz-dryrun.yaml— move the policy of the DENY filter (istio_authz_deny) fromrulestoshadow_rules(leave norules) and addshadow_rules_stat_prefix: istio_dry_run_deny_to the same filter. The ALLOW filter and the router stay as they are. Start Envoy again with this file, sendGET /admin/xandPOST /once each, and write four lines to/root/ist2-authz/07-dryrun.txt—<메서드> <경로>=<코드>for the two requests (the placeholders are the method, the path and the code),stat=(the full name of the statistic, among those ending inshadow_deniedin/statson the admin port, that has the prefix attached) andshadow_denied=(its value). - In
/root/ist2-authz/08-report.md, write five lines —chain=(the names of the step 3http_filtersin order, separated by commas),deny_body=(the body when RBAC denies),swapped_same=(thesame_as_04value of step 5),dry_run_field=(the name of the RBAC field the dry-run policy goes into) anddeny_tcp_fix=(the name of the field, recommended by the step 1 warning, to add to the operation of the DENY rule). Below them, write explanations starting with-in at least four lines.
Notes
- This Pod has neither a real istiod nor a real sidecar. So you cannot see the actual generated output with
istioctl proxy-config; instead you know the translation rules and build the equivalent Envoy configuration by hand to confirm the behavior. The same rules show up as they are in theproxy-configoutput of a production cluster. - The upstream imitation server knows only GET. If another method passes through the proxy, the app returns 501. A request the proxy blocked is told apart by the 403 and the body
RBAC: access denied. - In steps 4 to 7, you start Envoy again every time you change the configuration file. Do not edit an earlier step's file; make a new one under a new name — the grader looks at the files, not at the running Envoy.
- When you start Envoy, detach it completely from the shell with
setsid --fork nohup envoy -c <파일> --log-level warn > <로그> 2>&1 </dev/null(the placeholders are the file and the log). Before you start it again, clean up withpkill -x envoy(pkill -f 'envoy -c'also kills the shell itself that contains that string). - A server for imitating an upstream is already in the image:
python3 /opt/lab/envoy/upstream.py <포트> ok|fail|slow(the placeholder is the port). The response body is<모드>:<포트> <경로>(the placeholders are the mode, the port and the path). - After you edit the configuration, first filter it with
envoy --mode validate -c <파일>before you start it. The cluster name contains|, so in YAML you must always wrap it in quotes.
Write the two policies, DENY and ALLOW, and filter them with istioctl
Write an AuthorizationPolicy deny-admin to /root/ist2-authz/deny.yaml (namespace default, selector.matchLabels.app: payments, action: DENY, one rule — to.operation.paths: ["/admin*"]) and allow-get to /root/ist2-authz/allow.yaml (the same namespace and selector, action: ALLOW, to.operation.methods: ["GET"]). In /root/ist2-authz, put the output and exit code of istioctl validate -f deny.yaml -f allow.yaml in /root/ist2-authz/01-validate.txt (with rc= on the last line).
istioctl checks policies even without a cluster. A warning is attached to one of the two files — the exit code is 0, but it is content you must not just pass over. Read the warning sentence to the end. It is a warning that arises because the way DENY treats "if the request does not have that attribute" is the opposite of ALLOW, and you will use it again in step 8. Capture both streams of output with > 파일 2>&1 (the placeholder is the file).
Predict the result by the evaluation order before you start
With both policies applied to payments, predict by the evaluation order alone the verdicts for four requests — GET /, GET /admin/x, POST / and POST /admin — and write them to /root/ist2-authz/02-predict.txt as four lines of the form <메서드> <경로>=allow|deny (the placeholders are the method and the path; for example HEAD /=deny). You do not start anything yet.
Istio's evaluation order is five rows. If CUSTOM denies, deny → if it matches DENY, deny → if that workload has no ALLOW policy at all, allow → if it matches ALLOW, allow → everything else is denied. For each request, go down from the top and find the first row where it stops. /admin* is a prefix match. What matters is what happens to "a request that matched no ALLOW" on a workload that has an ALLOW policy.
Translate the two policies into two RBAC filters
Write an Envoy configuration to /root/ist2-authz/authz.yaml — admin port 9987; the listener virtualInbound is 127.0.0.1:10087, the HCM's stat_prefix is inbound_0.0.0.0_8110, and every path goes to the cluster inbound|8110|| (127.0.0.1:8110). http_filters has exactly three, in this order — (1) istio_authz_deny: RBAC, rules.action: DENY, policy name ns[default]-policy[deny-admin]-rule[0], permission is the url_path prefix /admin; (2) envoy.filters.http.rbac: RBAC, rules.action: ALLOW, policy name ns[default]-policy[allow-get]-rule[0], permission is the header :method exactly GET; both policies have principals: [{any: true}]; (3) the router. Then read http_filters with yq and write one line per filter to /root/ist2-authz/03-chain.txt as <이름> <rules.action> (the placeholder is the name; - for the router).
istiod gathers the DENY policies on one workload into one filter and the ALLOW policies into another, and puts the DENY one in front. This is because one RBAC filter can have only one action. The policy name ns[<네임스페이스>]-policy[<이름>]-rule[<번호>] (the placeholders are the namespace, the name and the number) is the form Istio actually attaches, so in production it is the key for finding the original resource from logs or config_dump. Istio's /admin* becomes url_path.path.prefix in Envoy and methods becomes a :method header match. "@type" is type.googleapis.com/envoy.extensions.filters.http.rbac.v3.RBAC. In yq, write the fallback for a missing value as (.a // "-").
Start it and compare with the prediction
Start an upstream with python3 /opt/lab/envoy/upstream.py 8110 ok and start Envoy with /root/ist2-authz/authz.yaml. Send the four requests from step 2 to localhost:10087 and write four lines of <메서드> <경로>=<HTTP 코드> (the placeholders are the method, the path and the HTTP code) to /root/ist2-authz/04-result.txt, and on the fifth line, body_403=, write the first line of the body that GET /admin/x received.
You change the method with curl -X POST. Get the code with -o 파일 -w '%{http_code}' (the placeholder is the file), and the body remains in that file. When the RBAC filter denies, the request does not reach the upstream and Envoy itself returns a 403 and a short body. Your prediction is right if the ones you said allow in step 2 give 200 and the ones you said deny give 403. If you were wrong, trace again which row it stopped at.
Even if you swap the filter order, the verdict is the same
Copy /root/ist2-authz/authz.yaml to /root/ist2-authz/authz-swapped.yaml, then change only the order of the two RBAC filters (the ALLOW filter in front, the DENY filter behind, the router at the very end, everything else as it is). Start Envoy again with this file, send the same four requests, and write four lines (<메서드> <경로>=<코드>, where the placeholders are the method, the path and the code) and same_as_04= (yes if all four codes are the same as in step 4, otherwise no) to /root/ist2-authz/05-swapped.txt.
In an HTTP filter chain, each RBAC filter can do only two things — pass to the next filter, or deny on the spot. There is no filter that skips the chain with a verdict of "allow". Then how are the conditions under which the two filters let a request through combined? You can swap the two list items with yq or edit the file by hand. Before you write down the result, filter it first with envoy --mode validate.
If there is no ALLOW policy at all, everything DENY did not catch passes
Make /root/ist2-authz/authz-denyonly.yaml from /root/ist2-authz/authz.yaml with only the ALLOW filter (envoy.filters.http.rbac) removed, and start Envoy again. Send POST / and GET /admin/x and write two lines (<메서드> <경로>=<코드>, where the placeholders are the method, the path and the code) to /root/ist2-authz/06-denyonly.txt, and on the third line, evaluation_rule=, write as a number which of the five rows of the evaluation order let POST / through.
If a workload has no ALLOW policy, istiod does not create the ALLOW filter at all — because if you put in an empty ALLOW filter, every request is denied (if the rules of RBAC has no policy at all, everything is denied). The upstream in this lab knows only GET, so it returns 501 for other methods. If it is not a 403 with RBAC: access denied, the proxy let it through, and the code is what the app decided. Write down the code you received as it is.
A dry-run puts the same policy into shadow_rules
Based on /root/ist2-authz/authz.yaml, make /root/ist2-authz/authz-dryrun.yaml — move the policy of the DENY filter (istio_authz_deny) from rules to shadow_rules (leave no rules) and add shadow_rules_stat_prefix: istio_dry_run_deny_ to the same filter. The ALLOW filter and the router stay as they are. Start Envoy again with this file, send GET /admin/x and POST / once each, and write four lines to /root/ist2-authz/07-dryrun.txt — <메서드> <경로>=<코드> for the two requests (the placeholders are the method, the path and the code), stat= (the full name of the statistic, among those ending in shadow_denied in /stats on the admin port, that has the prefix attached) and shadow_denied= (its value).
In Istio, if you put the annotation istio.io/dry-run: "true" on a policy, istiod puts that policy into shadow_rules instead of rules. The RBAC filter only evaluates the shadow side and does not use it in the verdict — the result is left only in statistics and dynamic metadata. An RBAC filter with no rules at all denies nothing. Where and with what separator the prefix is attached to the statistic name, see for yourself and copy: curl -s localhost:<관리포트>/stats | grep shadow (the placeholder is the admin port).
Summarize what an AuthorizationPolicy becomes in Envoy
In /root/ist2-authz/08-report.md, write five lines — chain= (the names of the step 3 http_filters in order, separated by commas), deny_body= (the body when RBAC denies), swapped_same= (the same_as_04 value of step 5), dry_run_field= (the name of the RBAC field the dry-run policy goes into) and deny_tcp_fix= (the name of the field, recommended by the step 1 warning, to add to the operation of the DENY rule). Below them, write explanations starting with - in at least four lines.
Copy the values from the files of the earlier steps. The warning in step 1 comes from DENY treating "an attribute the request does not have" as a match — a DENY that wrote only an HTTP path matches every TCP connection, which has no attribute called a path. Write the thing the warning sentence recommends narrowing by as the operation's field name (in the plural).