TT Lab
Get started
Learn Learning paths Courses

Istio Service Mesh

You opened an extension point and now upgrades are scary: EnvoyFilter and WasmPlugin

Continue in TT Lab

Goal

You check yourself how the API server enforces EnvoyFilter's four places (applyTo, match.context, patch.operation, and priority), and compare what changes when you write the same demand as a WasmPlugin. At the end you build a check script that scans all the EnvoyFilters in the cluster and prints version-upgrade risks as code.

Why it matters

If you use a mesh for a long time, demands that cannot be expressed in the standard API are sure to come. Then you end up using EnvoyFilter, and this API is not an Istio abstraction but directly touches Envoy's internal configuration structure. So when you upgrade Istio, the filter names or structure inside the proxy can change and the patch may quietly not be applied — no error appears and no alert rings. That said, telling people not to use it is not realistic. Instead, you turn three things into habits. Narrow the scope as much as possible (namespace and workloadSelector), write a priority on relative-position patches, and leave what you inserted where as a machine-readable checklist. Only then does a person not have to remember what to re-check before a version upgrade.

Steps

  1. Create /root/ist-envoyfilter and the namespace ext-lab. And write the two extension APIs registered in the cluster in two lines in /root/ist-envoyfilter/ext-apis.tsv as <복수형 이름>\t<API 그룹>\t<kind> (the placeholders are the plural resource name, the API group, and the kind) — they are EnvoyFilter and WasmPlugin. Sort them in ascending order of name.
  2. In /root/ist-envoyfilter/ef-broken.yaml, write an EnvoyFilter inbound-lua in the ext-lab namespace but get three places wrong on purpose — applyTo is HTTP_FILTERS, match.context is SIDECAR_IN, and patch.operation is INSERT_BEFORE_ALL. Put workloadSelector as app: checkout. Do not apply this file; do only a server-side trial application and save the rejection message to /root/ist-envoyfilter/ef-reject.txt.
  3. In /root/ist-envoyfilter/ef-inbound.yaml, write the fixed EnvoyFilter inbound-lua and actually apply it to ext-lab — applyTo is HTTP_FILTER, match.context is SIDECAR_INBOUND, patch.operation is INSERT_BEFORE, and match.listener.filterChain.filter.name is envoy.filters.network.http_connection_manager. In patch.value write only name: envoy.filters.http.lua, and do not put in priority yet.
  4. Create the namespace istio-system and apply in it an EnvoyFilter mesh-access-log — do not put a workloadSelector, write applyTo as NETWORK_FILTER, match.context as ANY, and patch.operation as MERGE, with a typed_config that attaches the access log to /dev/stdout. Keep the manifest in /root/ist-envoyfilter/ef-mesh.yaml. And for each EnvoyFilter currently in the cluster, write <네임스페이스>/<이름>\t<selector 있음 yes|no> (the placeholders are namespace/name and whether a selector exists, yes or no) sorted into /root/ist-envoyfilter/ef-scope.tsv.
  5. Save the output of istioctl analyze -n ext-lab -o json to /root/ist-envoyfilter/analyze-before.json — the inbound-lua you put up in step 3 must carry an IST0151 warning. Then write in /root/ist-envoyfilter/ef-inbound-priority.yaml a version with spec.priority added as 10 and apply it again, and save the output of the same command to /root/ist-envoyfilter/analyze-after.json. The latter must have no IST0151.
  6. In /root/ist-envoyfilter/wasm.yaml, write a WasmPlugin header-check in ext-lab and apply it — selector.matchLabels is app: checkout, url is oci://registry.lab.internal/plugins/header-check:1.0, phase is AUTHZ, priority is 20, and pluginConfig.header is x-lab-tier. And in /root/ist-envoyfilter/wasm-bad.yaml, write header-check-bad with phase: PRE_AUTHZ so that it is rejected by a server trial application, and save that message to /root/ist-envoyfilter/wasm-reject.txt.
  7. Run istioctl analyze -n ext-lab -o json again and write <코드>\t<대상> (the placeholders are the code and the target), one per line and sorted, in /root/ist-envoyfilter/findings.tsv. For the target, use the origin value of the analysis result as it is. And save the output of checking /root/ist-envoyfilter/ef-inbound.yaml (the file from step 3) with istioctl validate to /root/ist-envoyfilter/validate.txt — you confirm with your own eyes that what the analyzer catches and what the validator catches are not the same.
  8. Create the namespace legacy-lab and apply an EnvoyFilter legacy-ratelimit — no workloadSelector, applyTo is HTTP_FILTER, match.context is SIDECAR_INBOUND, patch.operation is INSERT_AFTER, and in patch.value write only name: envoy.filters.http.ratelimit (the manifest is /root/ist-envoyfilter/ef-legacy.yaml). Then create /root/ist-envoyfilter/ef-audit.sh so that it prints all the EnvoyFilters in the cluster as <네임스페이스>/<이름>\t<위험코드들> (the placeholders are namespace/name and the risk codes), sorted, only to standard output. Write the risk codes by joining the three of no-selector (no workloadSelector), no-priority (a relative-position operation without priority), and name-only (only a name without typed_config) with commas in dictionary order, and write - if there are none. Save the output to /root/ist-envoyfilter/audit-result.txt.

Notes

Check where the two extension APIs are registered

Create /root/ist-envoyfilter and the namespace ext-lab. And write the two extension APIs registered in the cluster in two lines in /root/ist-envoyfilter/ext-apis.tsv as <복수형 이름>\t<API 그룹>\t<kind> (the placeholders are the plural resource name, the API group, and the kind) — they are EnvoyFilter and WasmPlugin. Sort them in ascending order of name.

kubectl api-resources shows the plural name, the API group, and the kind. The two extension APIs are in different groups — one is in the same group as the traffic API, and the other is in an extension-dedicated group.

Get three enumerations wrong at once

In /root/ist-envoyfilter/ef-broken.yaml, write an EnvoyFilter inbound-lua in the ext-lab namespace but get three places wrong on purpose — applyTo is HTTP_FILTERS, match.context is SIDECAR_IN, and patch.operation is INSERT_BEFORE_ALL. Put workloadSelector as app: checkout. Do not apply this file; do only a server-side trial application and save the rejection message to /root/ist-envoyfilter/ef-reject.txt.

kubectl apply --dry-run=server only asks the API server and creates nothing. If the CRD has a structural schema, the server even returns the list of allowed values — count whether all three places come out at once.

Fix the three places and actually put it up

In /root/ist-envoyfilter/ef-inbound.yaml, write the fixed EnvoyFilter inbound-lua and actually apply it to ext-lab — applyTo is HTTP_FILTER, match.context is SIDECAR_INBOUND, patch.operation is INSERT_BEFORE, and match.listener.filterChain.filter.name is envoy.filters.network.http_connection_manager. In patch.value write only name: envoy.filters.http.lua, and do not put in priority yet.

To insert an HTTP filter, you must also point to which network filter's filter chain it is inside. That is why listener.filterChain.filter.name goes in match. priority is covered in the next step.

A place that applies to the whole mesh and a place that applies to one workload

Create the namespace istio-system and apply in it an EnvoyFilter mesh-access-log — do not put a workloadSelector, write applyTo as NETWORK_FILTER, match.context as ANY, and patch.operation as MERGE, with a typed_config that attaches the access log to /dev/stdout. Keep the manifest in /root/ist-envoyfilter/ef-mesh.yaml. And for each EnvoyFilter currently in the cluster, write <네임스페이스>/<이름>\t<selector 있음 yes|no> (the placeholders are namespace/name and whether a selector exists, yes or no) sorted into /root/ist-envoyfilter/ef-scope.tsv.

An EnvoyFilter placed in the root namespace (default istio-system) applies to the whole mesh. If placed in another namespace, only that namespace, and if you also attach a workloadSelector, only workloads whose labels match. The narrower the scope, the smaller the incident.

Without a priority on a relative-position patch, the analyzer warns

Save the output of istioctl analyze -n ext-lab -o json to /root/ist-envoyfilter/analyze-before.json — the inbound-lua you put up in step 3 must carry an IST0151 warning. Then write in /root/ist-envoyfilter/ef-inbound-priority.yaml a version with spec.priority added as 10 and apply it again, and save the output of the same command to /root/ist-envoyfilter/analyze-after.json. The latter must have no IST0151.

IST0151 appears when you use a relative-position operation like INSERT_BEFORE without writing a priority. It means the application order is not guaranteed, so the patch may not be reflected at all. If you add -o json, the code and target come out in a machine-readable form.

What changes when you write the same thing with the standard extension API

In /root/ist-envoyfilter/wasm.yaml, write a WasmPlugin header-check in ext-lab and apply it — selector.matchLabels is app: checkout, url is oci://registry.lab.internal/plugins/header-check:1.0, phase is AUTHZ, priority is 20, and pluginConfig.header is x-lab-tier. And in /root/ist-envoyfilter/wasm-bad.yaml, write header-check-bad with phase: PRE_AUTHZ so that it is rejected by a server trial application, and save that message to /root/ist-envoyfilter/wasm-reject.txt.

Instead of Envoy's internal structure, WasmPlugin writes only "what to insert at which stage." There are only four stage names, and the API server tells you the list. Also check that it is not created without a url.

Harden into a table what the analyzer catches in this namespace

Run istioctl analyze -n ext-lab -o json again and write <코드>\t<대상> (the placeholders are the code and the target), one per line and sorted, in /root/ist-envoyfilter/findings.tsv. For the target, use the origin value of the analysis result as it is. And save the output of checking /root/ist-envoyfilter/ef-inbound.yaml (the file from step 3) with istioctl validate to /root/ist-envoyfilter/validate.txt — you confirm with your own eyes that what the analyzer catches and what the validator catches are not the same.

jq -r '.[] | .code + "\t" + .origin' gives you a two-column table right away. validate looks at only one file, so it cannot catch what can be known only from cluster state. The two tools are not substitutes but checks at different points in time.

Build as a script the checklist to run before a version upgrade

Create the namespace legacy-lab and apply an EnvoyFilter legacy-ratelimit — no workloadSelector, applyTo is HTTP_FILTER, match.context is SIDECAR_INBOUND, patch.operation is INSERT_AFTER, and in patch.value write only name: envoy.filters.http.ratelimit (the manifest is /root/ist-envoyfilter/ef-legacy.yaml). Then create /root/ist-envoyfilter/ef-audit.sh so that it prints all the EnvoyFilters in the cluster as <네임스페이스>/<이름>\t<위험코드들> (the placeholders are namespace/name and the risk codes), sorted, only to standard output. Write the risk codes by joining the three of no-selector (no workloadSelector), no-priority (a relative-position operation without priority), and name-only (only a name without typed_config) with commas in dictionary order, and write - if there are none. Save the output to /root/ist-envoyfilter/audit-result.txt.

If you scan kubectl get envoyfilter -A -o json with jq, you can count everything at once. If you use a default-value operator like .spec.priority // "", 0 appears as missing, so compare with == null. The relative-position operations are the three INSERT_BEFORE, INSERT_AFTER, and INSERT_FIRST.