The Source the Partner's Log Saw

Goal

On a real Istio mesh, apply in turn the Sidecar scope, REGISTRY_ONLY, ServiceEntry, the egress gateway, and gateway authorization, and confirm from the receiving side's log and status codes what each one blocks and what it cannot block.

Why it matters

The default for outgoing traffic is 'pass to anywhere', so if you do nothing, the mesh does not know who goes where. Even if you apply the devices, if you do not distinguish whether it is a namespace setting or the gateway's enforcement, a gap remains where you believe 'it is blocked'. The habit of confirming with the source the receiving side (the partner) saw reveals that gap.

Steps

Put up the materials (the partner in outside outside the mesh, the client in shop inside the mesh, and the intruder in other) with kubectl apply -f /opt/fixtures/istlab/egress-app.yaml and wait until the Pods are ready. Then write three lines into /root/istlab-egress/01-baseline.txt — the status code of http://partner.outside/ from the client svc=, the status code of calling the partner Pod IP directly podip=, and the number of clusters the client sidecar knows clusters= (istioctl proxy-config clusters client.shop -o json | jq length).
Write the Sidecar default of the namespace shop into /root/istlab-egress/sidecar.yaml and apply it — the egress hosts are only two, ./* (the same namespace) and istio-system/*. After applying, write the client's number of clusters into /root/istlab-egress/02-scope.txt as clusters=.
Add outboundTrafficPolicy.mode: REGISTRY_ONLY to the Sidecar in /root/istlab-egress/sidecar.yaml and apply again. Then call the partner Pod IP from the client with an x-request-id attached, and save, exactly as it is, the one line of the client sidecar's access log that request left in /root/istlab-egress/03-blocked.log.
Write the ServiceEntry partner-api of the namespace shop into /root/istlab-egress/serviceentry.yaml and apply it — host api.partner.test, location: MESH_EXTERNAL, port 80 HTTP, resolution: DNS, endpoint address partner.outside.svc.cluster.local. After applying, write the address that api.partner.test resolves to from the client into /root/istlab-egress/04-se.txt as vip=, and the status code of http://api.partner.test/ as code=.
Write three resources into /root/istlab-egress/egress.yaml and apply them — (1) the Istio Gateway partner-egress of the namespace shop (selector istio: egressgateway, port 80, protocol HTTPS, tls.mode: ISTIO_MUTUAL, host api.partner.test), (2) the DestinationRule egressgateway-for-partner of istio-system (host istio-egressgateway.istio-system.svc.cluster.local, ISTIO_MUTUAL and sni: api.partner.test on port 80 of the subset partner), and (3) the VirtualService partner-via-egress of shop (host api.partner.test, gateways partner-egress and mesh — what comes from mesh goes to the egress gateway's subset partner 80, and what comes from the gateway goes to api.partner.test 80). After applying, write into /root/istlab-egress/05-path.txt the egress gateway Pod IP egress_pod_ip= and the source the partner log left for a request you sent with a marker attached to the path partner_saw=.
Call twice from the intruder in the other namespace and write into /root/istlab-egress/06-bypass.txt — the status code of calling http://partner.outside/ directly intruder_direct= and the status code of http://api.partner.test/ intruder_via_egress=. Also append the result of making the same direct call from the client as client_direct=.
Write the AuthorizationPolicy partner-only-shop of istio-system into /root/istlab-egress/egress-authz.yaml and apply it — selector istio: egressgateway, action: ALLOW, source principal cluster.local/ns/shop/sa/client, destination host api.partner.test. After applying, the client's api.partner.test must be 200 and the intruder's same request must be 403.
Write six lines into /root/istlab-egress/08-report.md — clusters_before= and clusters_after= (steps 1 and 2), blocked_code= (the status code of the step 3 log line), partner_saw= (egressgateway if the source the partner saw in step 5 is the egress gateway, client if it is the client), and intruder_direct= and intruder_via_egress= (step 6) — and write what you learned below that in at least four lines.

Notes

This VM takes 2–4 minutes to prepare. The sidecar's DNS proxy is on, so you can call the ServiceEntry host by name.
When you type commands on the client, specify the container, as in kubectl -n shop exec client -c curl -- curl ….
You look at the partner's log with kubectl -n outside logs partner and the gateway log with kubectl -n istio-system logs deploy/istio-egressgateway. In the sidecar log you find a request by attaching a request id, and since the gateway assigns a new id to requests that pass through it, you find them by attaching a marker to the path.
Right after you change the configuration, it takes a few seconds to spread to the proxies.
Common mistake — leaving the mesh gateway out of the VirtualService. Then the sidecar goes out directly without going through the egress gateway, and the client Pod IP is printed in the partner's log.

By default it goes out anywhere

Put up the materials (the partner in outside outside the mesh, the client in shop inside the mesh, and the intruder in other) with kubectl apply -f /opt/fixtures/istlab/egress-app.yaml and wait until the Pods are ready. Then write three lines into /root/istlab-egress/01-baseline.txt — the status code of http://partner.outside/ from the client svc=, the status code of calling the partner Pod IP directly podip=, and the number of clusters the client sidecar knows clusters= (istioctl proxy-config clusters client.shop -o json | jq length).

The mesh's default outside policy is ALLOW_ANY. A request going to a destination the sidecar does not know (a Pod IP that is not a Service, an external address) just flows out through the PassthroughCluster. And a sidecar by default knows every service in the cluster, so as services increase, every proxy's configuration grows with them. If you leave the two facts down as numbers, you can compare what changed in the later steps.

Reduce the range the proxy knows with the Sidecar resource

Write the Sidecar default of the namespace shop into /root/istlab-egress/sidecar.yaml and apply it — the egress hosts are only two, ./* (the same namespace) and istio-system/*. After applying, write the client's number of clusters into /root/istlab-egress/02-scope.txt as clusters=.

A Sidecar named default with no selector applies to every sidecar in that namespace. The egress hosts are the list of 'services this proxy needs to know', so services of namespaces not here drop out of the configuration — it is the main way to reduce proxy memory and istiod's push burden in a mesh with thousands of services. Also check that partner.outside is not in the reduced cluster list.

Block unknown destinations — REGISTRY_ONLY

Add outboundTrafficPolicy.mode: REGISTRY_ONLY to the Sidecar in /root/istlab-egress/sidecar.yaml and apply again. Then call the partner Pod IP from the client with an x-request-id attached, and save, exactly as it is, the one line of the client sidecar's access log that request left in /root/istlab-egress/03-blocked.log.

REGISTRY_ONLY means 'only to places in the service registry'. A destination the sidecar does not know goes to the BlackHoleCluster instead of the PassthroughCluster, and it is a 502. The registry of this namespace was reduced in step 2, so the partner.outside Service is now also 'an unknown place'. To find the request in the log, attach a request id yourself — kubectl -n shop logs client -c istio-proxy | grep <id>.

Put the partner in the registry — ServiceEntry

Write the ServiceEntry partner-api of the namespace shop into /root/istlab-egress/serviceentry.yaml and apply it — host api.partner.test, location: MESH_EXTERNAL, port 80 HTTP, resolution: DNS, endpoint address partner.outside.svc.cluster.local. After applying, write the address that api.partner.test resolves to from the client into /root/istlab-egress/04-se.txt as vip=, and the status code of http://api.partner.test/ as code=.

A ServiceEntry puts a destination outside the mesh in the registry so that it can go out even under REGISTRY_ONLY. But the name api.partner.test is not in the cluster DNS. This mesh has the sidecar's DNS proxy turned on, so the sidecar answers this name with a virtual IP it assigned automatically (240.240.x.x). That value is also in the ServiceEntry's status.addresses. Check with nslookup api.partner.test inside the Pod.

Gather the exit route into one egress gateway

Write three resources into /root/istlab-egress/egress.yaml and apply them — (1) the Istio Gateway partner-egress of the namespace shop (selector istio: egressgateway, port 80, protocol HTTPS, tls.mode: ISTIO_MUTUAL, host api.partner.test), (2) the DestinationRule egressgateway-for-partner of istio-system (host istio-egressgateway.istio-system.svc.cluster.local, ISTIO_MUTUAL and sni: api.partner.test on port 80 of the subset partner), and (3) the VirtualService partner-via-egress of shop (host api.partner.test, gateways partner-egress and mesh — what comes from mesh goes to the egress gateway's subset partner 80, and what comes from the gateway goes to api.partner.test 80). After applying, write into /root/istlab-egress/05-path.txt the egress gateway Pod IP egress_pod_ip= and the source the partner log left for a request you sent with a marker attached to the path partner_saw=.

A single VirtualService takes care of two segments. At the sidecar (mesh) it changes the destination to the egress gateway, and at the gateway it sends to the real destination. mTLS on the sidecar to gateway segment is not applied by itself — if you leave the Gateway as plain HTTP, the request passes but the gateway does not know the identity of the sender (you need it in step 7). You put the DestinationRule in istio-system so that the sidecars of other namespaces can also find the subset partner — if you put it in shop, only the sidecars of shop see it. Look at whether it was passed through from the receiving side. The gateway assigns a fresh x-request-id, so attach a marker to the path, as in http://api.partner.test/<표식> (the placeholder is the marker), find it in the partner log, and check whether that line's source is the egress gateway Pod IP.

A route that does not go through the gateway still remains

Call twice from the intruder in the other namespace and write into /root/istlab-egress/06-bypass.txt — the status code of calling http://partner.outside/ directly intruder_direct= and the status code of http://api.partner.test/ intruder_via_egress=. Also append the result of making the same direct call from the client as client_direct=.

The Sidecar resource and REGISTRY_ONLY are only the proxy configuration of shop. The proxies of other namespaces are still ALLOW_ANY, and a Pod without a sidecar does not know this configuration at all. The Istio docs also state that an egress gateway alone cannot force 'all outside traffic passes through the gateway' and that you must use another device such as a network policy together. Leave that gap down as a number.

Decide who can go out at the gateway

Write the AuthorizationPolicy partner-only-shop of istio-system into /root/istlab-egress/egress-authz.yaml and apply it — selector istio: egressgateway, action: ALLOW, source principal cluster.local/ns/shop/sa/client, destination host api.partner.test. After applying, the client's api.partner.test must be 200 and the intruder's same request must be 403.

From the sidecar to the egress gateway it is mTLS, so the gateway knows the identity (SPIFFE) of the workload that sent the request. So you can decide in a single place, the gateway, 'which namespace and service account can go out to this partner' — this is the real reason to have an egress gateway. If there is even one ALLOW policy, every request that does not match is rejected.

Write down where you blocked the exit route

Write six lines into /root/istlab-egress/08-report.md — clusters_before= and clusters_after= (steps 1 and 2), blocked_code= (the status code of the step 3 log line), partner_saw= (egressgateway if the source the partner saw in step 5 is the egress gateway, client if it is the client), and intruder_direct= and intruder_via_egress= (step 6) — and write what you learned below that in at least four lines.

Copy the values from the earlier step files. In the explanation lines, write in your own words 'what each of the Sidecar, REGISTRY_ONLY, the egress gateway, and the authorization policy blocks and what it cannot block'.