Watch What Cilium Actually Blocks

This lab runs on real Cilium

k3s + Cilium is actually running inside the VM. kube-proxy is turned off and Cilium does its job (kubeProxyReplacement), L7 policies are actually enforced by Envoy, and Hubble collects the flows.

The other labs in the CCA course run on a fake cluster with only the CRDs loaded. There, kubectl apply just passes and nothing is blocked. Here, things get blocked.

It takes 4–5 minutes to come up the first time, because Cilium is being installed.

Goal

See for yourself what an identity is, stack L3 and L7 policies one after another, and then read "what was blocked and why" with Hubble.

Why it matters

The basic Kubernetes NetworkPolicy also selects communication peers with label selectors on Pods and namespaces and with ipBlock. podSelector and namespaceSelector can be used in both from of ingress and to of egress. It is not an API where you have to list only IPs directly. The actual enforcement is done by a network plugin that supports it.

While implementing this policy intent, Cilium maps a set of security-relevant labels to an identity number. Sharing just app=tgt does not make it the same identity. Other security-relevant labels such as the namespace must also be compared. Endpoints in the same set share an identity, so you do not have to pin Pod IPs into policies one by one. Rather than treating the number itself as a permanent identifier, check which set of labels corresponds to that number.

And Cilium even looks at HTTP paths and methods. You can enforce "this service can call only /api/read" at the network layer, and when it is blocked, the connection is not cut; a 403 comes back — which is much easier for the application to handle.

Steps

1. Save cilium status to /root/cca/status.txt. KubeProxyReplacement: True must be visible, and there must be no kube-proxy DaemonSet.
2. In the mesh namespace, create tgt (app=tgt, nginx) and client (app=client, curl), and save the identity number of tgt and which labels that number came from in /root/cca/identity.txt.
3. With the l3-allow CiliumNetworkPolicy, let only app=client reach tgt, test both the allowed side and the side that is not, and save the result in /root/cca/l3.txt.
4. With the l7-allow policy, allow only the /allowed path. Record in /root/cca/l7.txt that /secret receives a 403.
5. After sending a request that gets blocked once, observe the flows with hubble observe and save them in /root/cca/hubble.txt. Both FORWARDED and DROPPED must be visible.
6. With the egress-fqdn policy, let tgt go out only to a specific name. Save the result in /root/cca/dnspolicy.txt.
7. With hubble observe --type policy-verdict, pull out the verdict log for why it was blocked and save it in /root/cca/verdict.txt.
8. In /root/cca/report.md, write the three lines tgt_identity=, l7_denied_code=, and policies= along with an explanation.

Notes

• The identity is in status.identity of kubectl get ciliumendpoint -n mesh tgt -o yaml. You can also see it with cilium identity list.
• Use Hubble like hubble observe -n mesh --last 30. You first need cilium hubble port-forward &.
• An FQDN policy works only when Cilium can look into the DNS responses. So to use toFQDNs, you must first open DNS itself with toEndpoints + rules.dns.matchPattern. If you leave this out, the names do not resolve at all and the policy does not work.
• Common mistake 1: using rules.http without toPorts in an L7 policy. HTTP rules go inside a port.
• Common mistake 2: expecting a connection failure when L7 blocked it. Cilium proxies through Envoy and returns a 403. It is not connection refused.

A cluster without kube-proxy

Save cilium status to /root/cca/status.txt. KubeProxyReplacement: True must be visible, and there must be no kube-proxy DaemonSet.

Look at the KubeProxyReplacement line of cilium status. If it is True, it means eBPF, not iptables, is doing the load balancing of Services.

Policies decide by number, not by IP

In the mesh namespace, create tgt (app=tgt, nginx) and client (app=client, curl), and save the identity number of tgt and which labels that number came from in /root/cca/identity.txt.

Look at status.identity in kubectl get ciliumendpoint -n mesh tgt -o yaml. Also include which combination of labels that number came from.

L3 — who can reach it

With the l3-allow CiliumNetworkPolicy, let only app=client reach tgt, test both the allowed side and the side that is not, and save the result in /root/cca/l3.txt.

endpointSelector selects the Pod being protected, and ingress.fromEndpoints selects the sources to allow. The direction rules are the same as in a Kubernetes NetworkPolicy.

L7 — how far along the path to allow

With the l7-allow policy, allow only the /allowed path. Record in /root/cca/l7.txt that /secret receives a 403.

Write the HTTP rules inside toPorts[].rules.http. When it is blocked, the connection is not cut; a 403 comes back.

See the flows with your own eyes

After sending a request that gets blocked once, observe the flows with hubble observe and save them in /root/cca/hubble.txt. Both FORWARDED and DROPPED must be visible.

Start cilium hubble port-forward & first and then use hubble observe -n mesh --last 30. You have to observe after sending the request that gets blocked for the DROPPED to show up.

Choose where traffic can go out by name

With the egress-fqdn policy, let tgt go out only to a specific name. Save the result in /root/cca/dnspolicy.txt.

toFQDNs works only when Cilium can look into the DNS responses. So you must first open DNS itself with toEndpoints + rules.dns.matchPattern.

Ask why it was blocked

With hubble observe --type policy-verdict, pull out the verdict log for why it was blocked and save it in /root/cca/verdict.txt.

hubble observe --type policy-verdict shows which policy each flow hit and how it was judged.

What did you learn

In /root/cca/report.md, write the three lines tgt_identity=, l7_denied_code=, and policies= along with an explanation.

Along with the three lines tgt_identity=, l7_denied_code=, and policies=, explain why an identity is better than an IP and why an L7 denial is a 403.