Having Written a Policy and Having It Enforced Are Different

In one line

A successful kubectl apply only means the policy was stored. Without a data plane, nothing gets blocked.

Why it has to be real Cilium

In the previous modules you wrote several CiliumNetworkPolicies. But the place where those labs ran was a cluster with only the CRDs loaded, so nothing was verified beyond the fact that kubectl apply succeeded.

In this module you set up real Cilium inside a VM and write the same policies again. Three things then become newly visible.

1. The identity is a real number

With kubectl get ciliumendpoint, you check the identity number together with the set of security-relevant labels. Even if one app label is the same, the identity can differ when other security labels such as the namespace differ. Endpoints with the same set of security labels share an identity. In a policy, rather than pinning a Pod IP that changes or a numeric ID that is not guaranteed to be permanent, you express the meaning of the labels you want to allow.

The basic Kubernetes NetworkPolicy also supports Pod and namespace selectors. It is not that only Cilium provides label policies; the points to compare are which data plane enforces the intent the API expresses, and how it extends the scope to things like FQDN and L7.

2. An L7 denial is a 403

For Cilium to see HTTP, it has to hand the packets to Envoy. So when traffic is blocked at L7, the connection is not cut; a 403 comes back.

This is better than an L3 block. A timeout cannot tell you "is the network strange, is the server dead, or is it a firewall?", but a 403 is an unambiguous denial. The client can immediately decide whether to retry, and the cause is left in the log as well.

3. Hubble tells you "why"

hubble observe --type policy-verdict shows which verdict each flow received in which direction. Without it, you end up deleting twenty policies one by one to find the culprit.

The order to check when a policy does not take effect

Sometimes you wrote a policy but nothing is blocked, or something that should not be blocked gets blocked. Check in this order.

1. Is the policy selecting that Pod?

kubectl get cep -n <ns> <pod> -o jsonpath='{.status.identity.labels}'
kubectl describe cnp <정책> | sed -n '/Endpoint Selector/,/Ingress/p'

The labels in the selector and the labels on the endpoint must match down to the last character. app: api and app: api are different.

2. Did you apply it in the right direction? A single communication has policies on both the outgoing side and the incoming side. You must allow both the egress at the source and the ingress at the destination for it to go through. The most common case is opening only one side and wandering in confusion over "I wrote a policy but it does not work."

3. Is default deny on? In Cilium, the moment even one policy selects that endpoint, the default for that direction changes to deny. If you attach a policy with only ingress rules, egress stays open as it is, but the moment you write a single egress rule, all the remaining egress is blocked. The typical case is forgetting DNS (port 53) so that even name lookup dies first.

egress:
  - toEndpoints:
      - matchLabels:
          k8s:io.kubernetes.pod.namespace: kube-system
          k8s:k8s-app: kube-dns
    toPorts:
      - ports: [{port: "53", protocol: UDP}]
        rules:
          dns: [{matchPattern: "*"}]

4. Look at the verdict with Hubble.

hubble observe --type policy-verdict --verdict DROPPED --last 20

The DROPPED line shows which policy blocked it, and in which direction. By the time you get here, you usually go back to one of 1–3.

The price of L7 policies

When you attach an L7 rule (rules.http), that traffic goes through Envoy. What you gain and what you lose are clear.

	L3/L4 policy	L7 policy
Where it is processed	eBPF (kernel)	Envoy (user space)
Added latency	Almost none	Hundreds of µs to a few ms
How it denies	Packet drop (timeout)	403 response
What it can see	IP and port	Method, path, headers

That is why you do not apply L7 to all communication. Put it only at the boundary where traffic enters from outside or in front of sensitive services, and keep L3/L4 between internal services.

What really matters in practice

Policies decide by identity, not by IP. So even if you restart a Pod and its IP changes, the policy does not waver, and the number of rules follows the number of identities, not the number of Pods. Label design is policy scale design.

It is a big gain in operations that an L7 denial arrives as a 403. A timeout cannot distinguish the network, the server, and the firewall, but a 403 is an unambiguous denial, so the client can immediately decide whether to retry, and the cause is left in the log as well.

When something is blocked, do not delete policies one by one; ask Hubble. hubble observe --type policy-verdict tells you which verdict was made in which direction. Without it, you end up binary-searching for the culprit in a bundle of twenty policies.

In the next lab you will confirm these three for yourself.