Kubernetes Networking — On a Real Cluster

Watch What a Policy Actually Blocks

This lab runs on real Kubernetes

A real single k3s node is running inside a VM, along with a controller that enforces NetworkPolicy. So when you attach a policy, traffic really is blocked. The grader also does not just look at files; each time it actually tries to connect to check.

It takes about 2 minutes to come up the first time.

Goal

Stack allowlist-style network policies one layer at a time, and in the process cause the most common incident in practice yourself and then fix it.

Why it matters

The Kubernetes default is allow everything. Any Pod in any namespace can reach every other Pod. If an intruder takes over one web Pod, the database is visible from right there.

But policies do not behave the way intuition says they should. If no policy is attached to a Pod, everything is allowed, and the moment even one is attached, that Pod switches to allowlist mode. So you end up writing "policies that allow" rather than "policies that block", and if you do not know about this switch, every policy you add cuts something unexpected.

The most common incident is step 5. The moment you default-deny egress, DNS is cut along with it. Port 53 going out to CoreDNS is also egress. The symptom is "the name cannot be found", so nobody suspects the network policy that was just attached.

Steps

What is ready: the shop namespace has api (app=api) and db (app=db, nginx) Pods, and there is an ops namespace. If it is missing, create it in step 1.

With no policy in place, confirm that any Pod can reach db and save it to /root/k8snp/baseline.txt. Response code 200 must be visible.
Create a default-deny policy in shop. It has podSelector: {} and policyTypes: [Ingress]. Then record in /root/k8snp/deny.txt that traffic is now blocked.
With an allow-api policy, let only the app=api Pods reach db. Save the result to /root/k8snp/allow-pod.txt. The policy attaches to the receiving side (app=db).
With an allow-ops policy, allow the whole ops namespace and save it to /root/k8snp/namespace.txt. Select the namespace by the kubernetes.io/metadata.name label.
Attach deny-egress (policyTypes: [Egress], empty selector) to shop and see what happens. Then write an incident report in /root/k8snp/incident.md. It must include that the cause is DNS, port 53, and which namespace CoreDNS is in.
Bring it back to life by opening only DNS with an allow-dns policy. Save the result to /root/k8snp/fixed.txt. You must open port 53 for both UDP and TCP.
Add a port restriction (80) to allow-api and save the result to /root/k8snp/ports.txt.
In /root/k8snp/report.md, write the two lines policies= (the number of policies) and dns_fix_policy=allow-dns, and explain the point where allow-everything flips to an allowlist.

Notes

Test connections with kubectl -n shop run t --rm -i --restart=Never --image=busybox:1.36 -- timeout 5 wget -q -O- http://<db의 IP>/ (the placeholder is the IP of db). If you use the Pod IP instead of a Service name, you can look at DNS problems and policy problems separately without mixing them.
Kubernetes attaches the kubernetes.io/metadata.name label to every namespace automatically. You do not need to attach it yourself.
In step 6, if you leave to empty it means "anywhere". Then DNS comes back, but all other egress is opened too and step 5 loses its meaning. Narrow it to the kube-system namespace.
Common mistake 1: attaching the policy to the sending side. An ingress rule always selects the Pod on the receiving side.
Common mistake 2: opening only UDP in step 6. When a DNS response exceeds 512 bytes, it moves to TCP. If you open only UDP, it fails only occasionally, which makes the cause much harder to find.
Common mistake 3: leaving ports empty in allow-api. Then api can reach all ports of db.

The default is allow everything

With no policy in place, confirm that any Pod can reach db and save it to /root/k8snp/baseline.txt. Response code 200 must be visible.

Measure the current state before you create a policy. To know what you changed, you need to know what it was before you changed it.

Attaching just one changes the mode

Create a default-deny policy in shop. It has podSelector: {} and policyTypes: [Ingress]. Then record in /root/k8snp/deny.txt that traffic is now blocked.

podSelector: {} means "every Pod in this namespace". If you give no rules at all, it becomes "nothing is allowed".

Attach it to the receiving side

With an allow-api policy, let only the app=api Pods reach db. Save the result to /root/k8snp/allow-pod.txt. The policy attaches to the receiving side (app=db).

The policy's podSelector selects the protected Pod (db), and ingress.from selects the source to allow (api). If you write the direction the other way around, nothing happens.

Allow by namespace

With an allow-ops policy, allow the whole ops namespace and save it to /root/k8snp/namespace.txt. Select the namespace by the kubernetes.io/metadata.name label.

namespaceSelector selects by the kubernetes.io/metadata.name label. Kubernetes attaches this label to every namespace automatically.

Block egress and names stop resolving

Attach deny-egress (policyTypes: [Egress], empty selector) to shop and see what happens. Then write an incident report in /root/k8snp/incident.md. It must include that the cause is DNS, port 53, and which namespace CoreDNS is in.

If you default-deny egress, port 53 going out to CoreDNS is blocked too. In the incident report, write the cause, the port and where CoreDNS is.

Open only DNS to bring it back

Bring it back to life by opening only DNS with an allow-dns policy. Save the result to /root/k8snp/fixed.txt. You must open port 53 for both UDP and TCP.

In to, select the kube-system namespace, and in ports put both UDP 53 and TCP 53.

Selecting only the source is half the job

Add a port restriction (80) to allow-api and save the result to /root/k8snp/ports.txt.

Add ports to the ingress entry. If you leave it empty, that source reaches every port of the target Pod.

What you learned

In /root/k8snp/report.md, write the two lines policies= (the number of policies) and dns_fix_policy=allow-dns, and explain the point where allow-everything flips to an allowlist.

Write the two lines policies= and dns_fix_policy=, and explain the point where allow-everything flips to an allowlist.