We Added a Waypoint and Got 503

Goal

On a real ambient mesh on k3s, experience separately the L4 enforced by the ztunnel and the L7 enforced by the waypoint, and confirm with requests and logs what you must fix together when you attach a waypoint.

Why it matters

Ambient is light because it removes sidecars, but policy and routing are split across two layers (ztunnel and waypoint). If you do not know which layer looks at what, an HTTP rule quietly drops out, traffic that was fine gets cut the moment you attach a waypoint, and a request coming by Pod IP bypasses the L7 policy. All three are invisible from the manifest alone, so they stay with you only if you experience them on a real mesh.

Steps

Put up the materials with kubectl apply -f /opt/fixtures/istlab/ambient-app.yaml, wait until the Pods are ready, and then attach the label istio.io/dataplane-mode=ambient to the namespace shop (you do not recreate the Pods). Then write two lines into /root/istlab-ambient/01-enroll.txt: web_containers= (the container names of the web-v1 Pod, separated by commas) and protocol= (the PROTOCOL column of the shop Pods in istioctl ztunnel-config workloads).
Call http://web/ once from the client, and save, exactly as it is, one ztunnel access log line that recorded that connection (the line whose message is connection complete and whose src.identity is client and dst.identity is web) in /root/istlab-ambient/02-ztunnel.log.
Write the AuthorizationPolicy web-l4 (namespace shop, selector app: web, ALLOW, principal cluster.local/ns/shop/sa/client) into /root/istlab-ambient/web-l4.yaml and apply it. After applying, call http://web.shop/ from the client and from the stranger outside, and write two lines into /root/istlab-ambient/03-l4.txt: client= (the status code) and stranger_exit= (the exit code of the stranger's curl).
Write the AuthorizationPolicy web-get-only-l4 (selector app: web, ALLOW, principal client, method GET) into /root/istlab-ambient/l7-wrong.yaml and apply it, and write the reason of the ZtunnelAccepted condition in that policy's status.conditions into /root/istlab-ambient/04-l7.txt as status_reason=, and the status code of the client's POST request as post_code=. After recording, delete this policy.
Attach a waypoint to shop with istioctl waypoint apply -n shop --enroll-namespace --wait. Right after that, write the status code of the client's http://web/ into /root/istlab-ambient/05-waypoint.txt as code_after_waypoint=, and fix the cause — change the allowed principal of /root/istlab-ambient/web-l4.yaml to only the waypoint's identity cluster.local/ns/shop/sa/waypoint and apply again. After the fix, the client's http://web/ must be 200, and if the client calls the web-v1 Pod IP directly, the connection must be dropped.
Write the AuthorizationPolicy web-get-only into /root/istlab-ambient/web-get-only.yaml and apply it — point at the Service web (group "", kind Service) with targetRefs, ALLOW, principal cluster.local/ns/shop/sa/client, method GET. After applying, the client's GET must be 200 and POST must be 403.
Write the HTTPRoute web into /root/istlab-ambient/httproute.yaml and apply it — the parent is the Service web (group "", kind Service, port 80), and there are two rules: if the header is x-canary: yes, web-v2, and otherwise web-v1. After applying, the client's request with x-canary: yes must be v2 and a request without the header must always be v1.
Write six lines into /root/istlab-ambient/08-report.md — protocol= (step 1), l7_without_waypoint= (enforced if the GET rule was enforced in step 4, ignored if it dropped out), broke_with_waypoint= (step 5's code_after_waypoint), l4_allowed_principal= (the principal web-l4 allows now), post_via_waypoint= (the status code of the client's POST now), and direct_pod_ip= (allowed or rejected if the client calls by Pod IP directly now) — and write what you learned below that in at least four lines.

Notes

This VM takes 2–4 minutes to prepare. The ambient profile (istiod, istio-cni, ztunnel) is installed with values.global.platform=k3s.
Ambient Pods have no sidecar, so you do not need to specify the container as in kubectl -n shop exec client -- curl ….
You look at the workloads and services the ztunnel knows with istioctl ztunnel-config workloads and istioctl ztunnel-config services, the ztunnel log with kubectl -n istio-system logs ds/ztunnel, and the waypoint log with kubectl -n shop logs deploy/waypoint.
Right after you change a policy, it takes a few seconds to spread to the ztunnel and waypoint.
Common mistake — writing HTTP conditions (method, path) in a selector policy. A waypoint policy is attached with targetRefs.

Put it in the mesh with a single label — without a restart

Put up the materials with kubectl apply -f /opt/fixtures/istlab/ambient-app.yaml, wait until the Pods are ready, and then attach the label istio.io/dataplane-mode=ambient to the namespace shop (you do not recreate the Pods). Then write two lines into /root/istlab-ambient/01-enroll.txt: web_containers= (the container names of the web-v1 Pod, separated by commas) and protocol= (the PROTOCOL column of the shop Pods in istioctl ztunnel-config workloads).

In ambient, the istio-cni node agent plants interception rules inside the Pod's network namespace, and the ztunnel, one per node, receives that traffic. You do not inject a sidecar, so there is no need to recreate the Pod. An enrolled Pod gets the annotation ambient.istio.io/redirection: enabled, and in the ztunnel's workload list the protocol changes to HBONE. The stranger outside the mesh stays TCP.

The two identities the ztunnel saw

Call http://web/ once from the client, and save, exactly as it is, one ztunnel access log line that recorded that connection (the line whose message is connection complete and whose src.identity is client and dst.identity is web) in /root/istlab-ambient/02-ztunnel.log.

Ambient's mTLS is established between ztunnels over HBONE (an mTLS tunnel over HTTP CONNECT, port 15008). The ztunnel is an L4 proxy, so it does not know the request path or method, but it knows the SPIFFE identities of the workloads on both sides. You look at the log with kubectl -n istio-system logs ds/ztunnel, and one connection can leave one line each on the sending side and the receiving side. The line with dst.hbone_addr attached is the connection that went in over HBONE.

The L4 policy the ztunnel enforces

Write the AuthorizationPolicy web-l4 (namespace shop, selector app: web, ALLOW, principal cluster.local/ns/shop/sa/client) into /root/istlab-ambient/web-l4.yaml and apply it. After applying, call http://web.shop/ from the client and from the stranger outside, and write two lines into /root/istlab-ambient/03-l4.txt: client= (the status code) and stranger_exit= (the exit code of the stranger's curl).

A policy attached with a selector is enforced by that Pod's ztunnel. The ztunnel looks at the source identity the moment it receives the connection and decides whether to allow it, so a rejection shows up not as an HTTP 403 but as a dropped connection (curl exit code 56). The stranger outside the mesh has no identity and matches no principal rule. You look at the exit code with …; echo $?.

What happens to an HTTP rule written without a waypoint

Write the AuthorizationPolicy web-get-only-l4 (selector app: web, ALLOW, principal client, method GET) into /root/istlab-ambient/l7-wrong.yaml and apply it, and write the reason of the ZtunnelAccepted condition in that policy's status.conditions into /root/istlab-ambient/04-l7.txt as status_reason=, and the status code of the client's POST request as post_code=. After recording, delete this policy.

The ztunnel does not interpret HTTP. If you give a rule with conditions such as method or path as a selector policy, istiod sends it down to the ztunnel with that rule left out and notes that fact in the policy's status. Among ALLOW policies, it is 'allow if even one matches', so if the existing web-l4 already allows the client, POST also passes — you wrote that only GET is allowed, yet POST works. You look at the status with kubectl -n shop get authorizationpolicy web-get-only-l4 -o yaml.

After attaching a waypoint, a request that was fine is 503

Attach a waypoint to shop with istioctl waypoint apply -n shop --enroll-namespace --wait. Right after that, write the status code of the client's http://web/ into /root/istlab-ambient/05-waypoint.txt as code_after_waypoint=, and fix the cause — change the allowed principal of /root/istlab-ambient/web-l4.yaml to only the waypoint's identity cluster.local/ns/shop/sa/waypoint and apply again. After the fix, the client's http://web/ must be 200, and if the client calls the web-v1 Pod IP directly, the connection must be dropped.

Once a waypoint exists, a request going to a service goes in the order client's ztunnel, then the waypoint, then web's ztunnel. So the source web's ztunnel sees is not the client but the waypoint. The L4 policy that allowed only the client cuts that connection, and the waypoint returns a 503 (tunnel_response:401 in the waypoint log). If you allow only the waypoint identity, the route to the service opens and the route that does not go through the waypoint (directly by Pod IP) closes — because a service waypoint does not handle a request that came by Pod IP, you have to close it like this so that you do not bypass the L7 policy of step 7.

Put the HTTP rules on the waypoint

Write the AuthorizationPolicy web-get-only into /root/istlab-ambient/web-get-only.yaml and apply it — point at the Service web (group "", kind Service) with targetRefs, ALLOW, principal cluster.local/ns/shop/sa/client, method GET. After applying, the client's GET must be 200 and POST must be 403.

A waypoint is an Envoy, so it interprets HTTP. The way to attach a policy to a waypoint is not a selector but targetRefs — if you point at a Service, the waypoint that handles requests going to that Service enforces it. The source the waypoint sees is the client itself, so you write the principal as the client. This time the rejection is not a dropped connection but an HTTP 403 (RBAC: access denied).

The waypoint also takes care of routing — HTTPRoute

Write the HTTPRoute web into /root/istlab-ambient/httproute.yaml and apply it — the parent is the Service web (group "", kind Service, port 80), and there are two rules: if the header is x-canary: yes, web-v2, and otherwise web-v1. After applying, the client's request with x-canary: yes must be v2 and a request without the header must always be v1.

In ambient, in-mesh routing is also written with the Gateway API. What differs from ingress is that the parent is a Service, not a Gateway (GAMMA). What actually handles this route is web's waypoint, so without a waypoint the route is not applied. You see whether it attached through the ResolvedWaypoints condition in the HTTPRoute status.

Sort out who handles L4 and L7

Write six lines into /root/istlab-ambient/08-report.md — protocol= (step 1), l7_without_waypoint= (enforced if the GET rule was enforced in step 4, ignored if it dropped out), broke_with_waypoint= (step 5's code_after_waypoint), l4_allowed_principal= (the principal web-l4 allows now), post_via_waypoint= (the status code of the client's POST now), and direct_pod_ip= (allowed or rejected if the client calls by Pod IP directly now) — and write what you learned below that in at least four lines.

Look at the records of the earlier steps together with the current state. In the explanation lines, write 'what the ztunnel and the waypoint each look at and enforce' and 'why you must fix the L4 policy together when you attach a waypoint'.