Insert a patch and tell istioctl's verdict from Envoy's

Goal

Put a fault filter before router with an EnvoyFilter, and reproduce by hand where that patch ends up in the Envoy configuration. You make one that is put after router and one with a wrong value, see the verdicts of istioctl and Envoy part ways, and confirm version pinning and scope of application.

Why it matters

EnvoyFilter is the most powerful resource in the mesh and also the one that most easily breaks quietly. If Envoy rejects what istioctl passed, in production it shows only as "the configuration does not change". If you know where a patch lands and the limits of the checking, you can filter it out before merging.

Steps

Create /root/ist2-ef and write an EnvoyFilter to /root/ist2-ef/ef.yaml — apiVersion: networking.istio.io/v1alpha3, name reviews-fault, namespace default, workloadSelector app: reviews. There is one patch: applyTo: HTTP_FILTER, match.context: SIDECAR_INBOUND, match.listener.filterChain.filter.name is envoy.filters.network.http_connection_manager, its subFilter.name is envoy.filters.http.router, and patch.operation: INSERT_BEFORE. The value to insert is the name envoy.filters.http.fault with a typed_config (@type is type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault) holding abort.http_status: 418, and abort.percentage is numerator: 100 and denominator: HUNDRED. Put the output of istioctl validate -f /root/ist2-ef/ef.yaml (including standard error) and its exit code in /root/ist2-ef/01-validate.txt (with rc= on the last line).
From the first patch of /root/ist2-ef/ef.yaml, pull out five values with yq and write them to /root/ist2-ef/02-fields.txt — applyTo=, context= (match.context), operation= (patch.operation), anchor= (the HTTP filter match uses as its reference, that is subFilter.name) and filter= (the name of the value being inserted).
Write an Envoy configuration to /root/ist2-ef/envoy-before.yaml — admin port 9989, a listener virtualInbound listening at 127.0.0.1:10089, the HTTP connection manager's stat_prefix is inbound_0.0.0.0_9080, and every path goes to the cluster inbound|9080|| (127.0.0.1:8112). http_filters is, in this order, a fault filter entry exactly the same as the patch value in ef.yaml, and then router. Start an upstream on 8112 as ok and start Envoy, then send curl localhost:10089/reviews three times and write three lines to /root/ist2-ef/03-before.txt — codes= (the three response codes, separated by commas), stat_name= (the full name of the statistic that counts the requests fault aborted) and aborts_injected= (the value of that statistic).
Copy /root/ist2-ef/ef.yaml to /root/ist2-ef/ef-after.yaml and change only patch.operation to INSERT_AFTER. And copy /root/ist2-ef/envoy-before.yaml to /root/ist2-ef/envoy-after.yaml and flip only the order of http_filters to router, fault (the result with that patch applied). Check the two files with istioctl validate -f and envoy --mode validate -c respectively and write to /root/ist2-ef/04-after.txt — the first line istioctl_rc=, the second line envoy_rc=, and below that, as it is, the output line with the reason Envoy rejected it.
Copy /root/ist2-ef/ef.yaml to /root/ist2-ef/ef-badfield.yaml and /root/ist2-ef/envoy-before.yaml to /root/ist2-ef/envoy-badfield.yaml, then change the fault abort.percentage in both files to the shape used in a VirtualService, { value: 100 } (leave everything else as it is). Check with istioctl validate -f and envoy --mode validate -c respectively and write to /root/ist2-ef/05-gap.txt — the first line istioctl_rc=, the second line envoy_rc=, and below that, as they are, the warning line where istioctl objected to this field and the reason line where Envoy rejected it.
Copy /root/ist2-ef/ef.yaml to /root/ist2-ef/ef-pinned.yaml and add the regular expression ^1\.24.* to match.proxy.proxyVersion of the first patch (leave everything else as it is). Confirm it passes with istioctl validate, and to find out the proxy version in this lab, inject /opt/lab/fixtures/istio/inject-target.yaml with istioctl kube-inject in /root/ist2-ef and save it as /root/ist2-ef/inject.yaml (pass all three injection configuration files /opt/istio/inject-config.yaml, mesh-config.yaml and values-config.yaml). Write four lines to /root/ist2-ef/06-version.txt — regex= (the regular expression you wrote in ef-pinned.yaml, as it is), proxy_version= (the tag of the istio-proxy image), matches_proxy= (yes if that version matches the regular expression, otherwise no) and matches_1_25_0= (yes if a hypothetical version 1.25.0 matches, otherwise no).
Write an EnvoyFilter to /root/ist2-ef/ef-ratelimit.yaml — name inbound-ratelimit, namespace default, with no workloadSelector, the patch shape the same as ef.yaml (HTTP_FILTER, SIDECAR_INBOUND, INSERT_BEFORE before router), and the value is the name envoy.filters.http.local_ratelimit, @type type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit, stat_prefix: http_local_rate_limiter, token_bucket with max_tokens: 1, tokens_per_fill: 1 and fill_interval: 300s, and both filter_enabled and filter_enforced have default_value of numerator: 100 and denominator: HUNDRED. After you confirm with istioctl validate that there is no value warning, write to /root/ist2-ef/envoy-rl.yaml an Envoy configuration with that value placed before router (the admin port, listener and cluster are the same as step 3). Start it and send curl localhost:10089/reviews three times, then write four lines to /root/ist2-ef/07-ratelimit.txt — codes= (the three response codes, separated by commas), rate_limited= (the value of the statistic http_local_rate_limit.rate_limited), scope= (the scope this EnvoyFilter reaches: one of workload, namespace and mesh) and mesh_wide_namespace= (the namespace where, if you move the same file there, it reaches the whole mesh, from /opt/istio/mesh-config.yaml).
In /root/ist2-ef/08-report.md, write four lines, before_status=, after_envoy_rc=, badfield_istioctl_rc= and ratelimit_codes= (respectively the status code fault returned in step 3, the exit code of Envoy in step 4, the exit code of istioctl in step 5, and the three response codes of step 7), and below them write explanations starting with - in at least four lines.

Notes

This Pod has neither a real istiod nor a real sidecar. So you cannot see the actual generated output with istioctl proxy-config; instead you know the translation rules and build the equivalent Envoy configuration by hand to confirm the behavior. The same rules show up as they are in the proxy-config output of a production cluster.
istioctl validate gives the warning "exposes internal implementation details" even for a correct EnvoyFilter. This warning is normal; see whether a separate warning that objects to a field inside the value also appears. Warnings come out on standard error too, so attach 2>&1 when you put them in a file.
Envoy's error wording deliberately has wobbling spaces between words (no such field). Do not be surprised when you read it by eye.
The configuration file names differ for each step. Copy and use the files of earlier steps; do not edit them — the grading of the earlier steps looks at those files.
When you start Envoy, detach it completely from the shell with setsid --fork nohup envoy -c <파일> --log-level warn > <로그> 2>&1 </dev/null (the placeholders are the file and the log). Before you start it again, clean up with pkill -x envoy (pkill -f 'envoy -c' also kills the shell itself that contains that string).
A server for imitating an upstream is already in the image: python3 /opt/lab/envoy/upstream.py <포트> ok|fail|slow (the placeholder is the port). The response body is <모드>:<포트> <경로> (the placeholders are the mode, the port and the path).
After you edit the configuration, first filter it with envoy --mode validate -c <파일> before you start it. The cluster name contains |, so in YAML you must always wrap it in quotes.

Write an EnvoyFilter that puts fault before router

Create /root/ist2-ef and write an EnvoyFilter to /root/ist2-ef/ef.yaml — apiVersion: networking.istio.io/v1alpha3, name reviews-fault, namespace default, workloadSelector app: reviews. There is one patch: applyTo: HTTP_FILTER, match.context: SIDECAR_INBOUND, match.listener.filterChain.filter.name is envoy.filters.network.http_connection_manager, its subFilter.name is envoy.filters.http.router, and patch.operation: INSERT_BEFORE. The value to insert is the name envoy.filters.http.fault with a typed_config (@type is type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault) holding abort.http_status: 418, and abort.percentage is numerator: 100 and denominator: HUNDRED. Put the output of istioctl validate -f /root/ist2-ef/ef.yaml (including standard error) and its exit code in /root/ist2-ef/01-validate.txt (with rc= on the last line).

One patch of an EnvoyFilter is a pair of "where" (applyTo and match) and "what" (operation and value). If you pick a reference filter with subFilter on HTTP_FILTER, the filter list of the HTTP connection manager that contains that filter becomes the stage. value is not Istio syntax but Envoy configuration as it is, so a percentage too must be written as Envoy's FractionalPercent (numerator and denominator). Even for a correct file, istioctl gives one warning line about EnvoyFilter itself — that warning is normal, and if more warnings appear that object to a field inside the value, you have to fix them.

Read the "where" and "what" of a patch as four fields

From the first patch of /root/ist2-ef/ef.yaml, pull out five values with yq and write them to /root/ist2-ef/02-fields.txt — applyTo=, context= (match.context), operation= (patch.operation), anchor= (the HTTP filter match uses as its reference, that is subFilter.name) and filter= (the name of the value being inserted).

applyTo is the kind of Envoy object the patch touches (listener, filter chain, network filter, HTTP filter, cluster, route …), context is which direction of proxy it is (the sidecar's incoming side or outgoing side, or a gateway), and match is the condition that picks one within that kind. A relative-position operation such as INSERT_BEFORE has meaning only if this reference filter exists. Do not mix up filter.name and subFilter.name — the former is a network filter and the latter is an HTTP filter.

Set up the patched result in Envoy and get a 418

Write an Envoy configuration to /root/ist2-ef/envoy-before.yaml — admin port 9989, a listener virtualInbound listening at 127.0.0.1:10089, the HTTP connection manager's stat_prefix is inbound_0.0.0.0_9080, and every path goes to the cluster inbound|9080|| (127.0.0.1:8112). http_filters is, in this order, a fault filter entry exactly the same as the patch value in ef.yaml, and then router. Start an upstream on 8112 as ok and start Envoy, then send curl localhost:10089/reviews three times and write three lines to /root/ist2-ef/03-before.txt — codes= (the three response codes, separated by commas), stat_name= (the full name of the statistic that counts the requests fault aborted) and aborts_injected= (the value of that statistic).

When istiod applies the EnvoyFilter, the HTTP filter list on this sidecar's incoming listener becomes [..., fault, router]. There is no istiod in this Pod, so you make that result by hand. The key point is that the patch's value becomes as it is one entry of the list — do not change the value as you carry it over. HTTP filter statistics accumulate under http.<stat_prefix>.. Narrow them down with /stats?filter= on the admin port.

Put it after router and istioctl passes, but Envoy rejects

Copy /root/ist2-ef/ef.yaml to /root/ist2-ef/ef-after.yaml and change only patch.operation to INSERT_AFTER. And copy /root/ist2-ef/envoy-before.yaml to /root/ist2-ef/envoy-after.yaml and flip only the order of http_filters to router, fault (the result with that patch applied). Check the two files with istioctl validate -f and envoy --mode validate -c respectively and write to /root/ist2-ef/04-after.txt — the first line istioctl_rc=, the second line envoy_rc=, and below that, as it is, the output line with the reason Envoy rejected it.

router is the filter that sends the request to the upstream and ends it, so Envoy blocks a filter coming after it at the configuration stage. istioctl knows only the outer structure of an EnvoyFilter and does not know Envoy's filter order rules, so it lets this patch through. In production, istiod pushes it in as it is and Envoy rejects that listener update (a NACK) — it looks like the Pod seems fine but the configuration does not change. Get the exit code from $? right after the command.

Copy over a VirtualService-style percentage and only a warning comes out

Copy /root/ist2-ef/ef.yaml to /root/ist2-ef/ef-badfield.yaml and /root/ist2-ef/envoy-before.yaml to /root/ist2-ef/envoy-badfield.yaml, then change the fault abort.percentage in both files to the shape used in a VirtualService, { value: 100 } (leave everything else as it is). Check with istioctl validate -f and envoy --mode validate -c respectively and write to /root/ist2-ef/05-gap.txt — the first line istioctl_rc=, the second line envoy_rc=, and below that, as they are, the warning line where istioctl objected to this field and the reason line where Envoy rejected it.

The fault.abort.percentage.value of a VirtualService is Istio's percentage type, and Envoy's fault filter uses FractionalPercent, made of a numerator and a denominator. It is a mistake people often make when copying over. istioctl does try to unpack the value into an Envoy type, but it reports an unknown field only as a warning and the exit code is 0 — if CI looks only at the exit code, it is deployed as it is. On the other hand, if the @type name itself is wrong, istioctl blocks it with an error too. This is where the criterion for which to trust splits. If you change only one field with yq, you do not touch the rest.

Tie a patch to one version with proxyVersion

Copy /root/ist2-ef/ef.yaml to /root/ist2-ef/ef-pinned.yaml and add the regular expression ^1\.24.* to match.proxy.proxyVersion of the first patch (leave everything else as it is). Confirm it passes with istioctl validate, and to find out the proxy version in this lab, inject /opt/lab/fixtures/istio/inject-target.yaml with istioctl kube-inject in /root/ist2-ef and save it as /root/ist2-ef/inject.yaml (pass all three injection configuration files /opt/istio/inject-config.yaml, mesh-config.yaml and values-config.yaml). Write four lines to /root/ist2-ef/06-version.txt — regex= (the regular expression you wrote in ef-pinned.yaml, as it is), proxy_version= (the tag of the istio-proxy image), matches_proxy= (yes if that version matches the regular expression, otherwise no) and matches_1_25_0= (yes if a hypothetical version 1.25.0 matches, otherwise no).

istiod checks the version the proxy reported about itself when it connected (the ISTIO_VERSION metadata) against this regular expression, and attaches the patch only when it matches. If an upgrade changes Envoy's filter names or configuration shape, an old patch can break a new proxy, but if you tie the version, the patch is not attached at all to the new proxy. A dot in a regular expression is any character, so guard it with \., and anchor the front with ^. You can try the matching with grep -E. Because of the empty document at the end of the injection output, filter yq with select(.kind=="Deployment").

Put a request limit on a whole namespace with an EnvoyFilter that has no selector

Write an EnvoyFilter to /root/ist2-ef/ef-ratelimit.yaml — name inbound-ratelimit, namespace default, with no workloadSelector, the patch shape the same as ef.yaml (HTTP_FILTER, SIDECAR_INBOUND, INSERT_BEFORE before router), and the value is the name envoy.filters.http.local_ratelimit, @type type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit, stat_prefix: http_local_rate_limiter, token_bucket with max_tokens: 1, tokens_per_fill: 1 and fill_interval: 300s, and both filter_enabled and filter_enforced have default_value of numerator: 100 and denominator: HUNDRED. After you confirm with istioctl validate that there is no value warning, write to /root/ist2-ef/envoy-rl.yaml an Envoy configuration with that value placed before router (the admin port, listener and cluster are the same as step 3). Start it and send curl localhost:10089/reviews three times, then write four lines to /root/ist2-ef/07-ratelimit.txt — codes= (the three response codes, separated by commas), rate_limited= (the value of the statistic http_local_rate_limit.rate_limited), scope= (the scope this EnvoyFilter reaches: one of workload, namespace and mesh) and mesh_wide_namespace= (the namespace where, if you move the same file there, it reaches the whole mesh, from /opt/istio/mesh-config.yaml).

If there is a workloadSelector, it attaches only to workloads whose labels match; if there is none, to all workloads in that namespace; and if you put it in the root namespace of the mesh configuration, to the whole mesh. Those in the root namespace are applied first and those in the workload namespace later. Do not put runtime_key inside the request limit value — the Envoy of this version rejects it. If there is one token and the refill interval is long, only the first request gets through. See for yourself what the code of the limited response is.

Summarize it as a checklist for before you use an EnvoyFilter

In /root/ist2-ef/08-report.md, write four lines, before_status=, after_envoy_rc=, badfield_istioctl_rc= and ratelimit_codes= (respectively the status code fault returned in step 3, the exit code of Envoy in step 4, the exit code of istioctl in step 5, and the three response codes of step 7), and below them write explanations starting with - in at least four lines.

Copy the values from the files of the earlier steps. If in the explanation lines you write "what do I check before merging an EnvoyFilter", this table becomes a review checklist — where the patch lands, the position of router, what istioctl cannot catch, version pinning and scope.