TT Lab
Get started
Learn Learning paths Courses

Kubernetes Networking — On a Real Cluster

There Is No "Deny" in a Policy

Continue in TT Lab

In one line

NetworkPolicy has no concept of deny. If a Pod has no policy at all, everything is allowed, and the moment even one policy attaches, that direction flips to an allowlist.

Why you need to know about this switch

The Kubernetes default is allow everything. Any Pod in any namespace can reach every other Pod. If an intruder takes over one web Pod, the database is visible from right there.

Where the default flips

NetworkPolicy does not behave the way intuition says it should.

If no policy is attached to a Pod, everything is allowed, and the moment even one is attached, that Pod switches to allowlist mode for that direction.

So a policy has no concept of "deny" at all. A policy named default-deny is really just a policy that allows nothing.

spec:
  podSelector: {}          # 이 네임스페이스의 모든 파드
  policyTypes: [Ingress]   # ingress 만 다룬다
  # ingress: 항목이 없다 → 허용하는 것이 하나도 없다

If you do not know about this switch, you end up with "I added just one policy and something unrelated broke".

How not to mix up the direction

A policy is always attached to the receiving side.

Field What it selects
podSelector The protected Pod (db)
ingress.from The source to allow (api)

If you attach it to the sending side, the syntax passes and the object is created, but nothing happens. It is more confusing because there is no error.

Multiple policies add up

When several policies are attached to the same Pod, the result is their union. A policy added later does not overwrite an earlier one. That is why you can end up "adding a policy to narrow things, and it gets wider". To narrow, you have to edit the existing policy.

The most common incident

The moment you default-deny egress, DNS is cut along with it. Port 53 going out to CoreDNS is also egress.

The symptom is "the name cannot be found", CoreDNS is fine, and other namespaces are normal. So nobody suspects the network policy that was just attached.

The order for introducing your first policies

If you turn on default deny first, everything is cut that instant. There is an order to follow.

1. Look at what is communicating first. Before you write a policy, look at the real flows. With Cilium use Hubble; otherwise map them out from conntrack or application logs.

hubble observe --namespace labhub-prod --last 500   -o json | jq -r '"\(.source.namespace)/\(.source.pod_name) → \(.destination.namespace)/\(.destination.pod_name):\(.l4.TCP.destination_port)"'   | sort | uniq -c | sort -rn

2. Put the allow policies in first. There is no default deny yet, so nothing is blocked.

3. Then turn on default deny. This is when what you missed shows up.

4. Always check DNS. The moment you use egress, DNS (53/UDP) is blocked too. If name lookup fails, everything looks like a timeout and the cause is hard to find.

egress:
  - to:
      - namespaceSelector:
          matchLabels: {kubernetes.io/metadata.name: kube-system}
        podSelector:
          matchLabels: {k8s-app: kube-dns}
    ports:
      - {protocol: UDP, port: 53}
      - {protocol: TCP, port: 53}

What policies cannot block

Network policy is not all-powerful. You need to know what it cannot block so that you send those cases to another layer.

What it cannot block Use instead
Containers within the same Pod Split the Pod
hostNetwork Pods Forbid hostNetwork with Pod Security
Traffic leaving the node itself Node firewall, security groups
L7 (path, method) Cilium L7 policy, service mesh
Connections already established A policy applies only to new connections

The last row is the trap. Even after you add a policy, connections that were already in progress stay alive. That is how you get "I added the policy, but it still gets through". To check, you have to open a new connection.

Debugging order

# 1. 이 파드를 고르는 정책이 있나 (없으면 기본 허용)
kubectl get netpol -n <ns> -o json | jq -r '
  .items[] | select(.spec.podSelector.matchLabels // {} | to_entries | length > 0) |
  "\(.metadata.name): \(.spec.podSelector.matchLabels) \(.spec.policyTypes)"'

# 2. 양쪽 다 열려 있나 — 출발지 egress, 도착지 ingress
# 3. 실제로 막혔는지 확인 (Connection refused 는 정책이 아니다)
kubectl exec -n <ns> <pod> -- nc -zv -w3 <대상> <포트>

The distinction in item 3 matters. If a policy blocks it, you get a timeout, and if nobody is listening, you get refused. If you see refused, it is not a policy problem.

What really matters in practice

When you block egress, open DNS in the same commit. Port 53 going out to CoreDNS is also egress, so it is cut too, yet the symptom is "the name cannot be found", CoreDNS is fine, and other namespaces are normal. So nobody suspects the policy that was just attached.

Always attach a policy to the receiving side. If you attach it to the sending side, the syntax passes and the object is created, but nothing happens. Because there is no error, you wander around longer.

To narrow, edit the existing policy instead of adding one. Policies attached to the same Pod are a union, so a later one does not overwrite an earlier one. This is where "I added one to narrow things, and it got wider" comes from.

In the next lab you cause this incident yourself and then fix it. You will also see why opening only UDP 53 when you fix it leaves things in a worse state.