The certificate was renewed, but the browser still showed the old one

The authority changed, but the cache serves the old answer

Goal

Start an authoritative DNS server and a cache server in Python, read the TTL, CNAME, and negative responses with dig, and then reproduce for yourself the situation "the zone was changed, but the cache still gives the old answer."

Why it matters

Most DNS incidents are a question of "who remembers what, and for how long." /etc/hosts answers before DNS does, the cache returns the old answer for as long as the TTL the authoritative server gave it, and even a nonexistent name is remembered for the time the SOA sets. In this lab you work with each of those three layers by hand. The provided server /opt/app/minidns.py is a small DNS built with the standard library: in authoritative mode it rereads the zone file on every query, and in cache mode it answers with the TTL counting down.

Steps

Create the zone file /root/dns/zone.txt. It must contain the SOA for lab.internal. (the last field, minimum, is 60), an A record www.lab.internal. → 10.0.0.10 with TTL 120, and a CNAME api.lab.internal. → www.lab.internal.. Start the authoritative server with python3 /opt/app/minidns.py auth --zone /root/dns/zone.txt --port 5300 (send its log to /root/dns/auth.log), and save the output of dig @127.0.0.1 -p 5300 www.lab.internal A +noall +answer to /root/dns/01-auth.txt.
Add 10.0.0.99 www.lab.internal to /etc/hosts, and save the output of getent hosts www.lab.internal together with the hosts: line of /etc/nsswitch.conf to /root/dns/02-hosts.txt.
Start the cache server with python3 /opt/app/minidns.py cache --upstream 127.0.0.1:5300 --port 5301, query the cache (5301) for www.lab.internal twice, a few seconds apart, and save both answers to /root/dns/03-ttl.txt.
Ask the cache for the A record of api.lab.internal and save the answer to /root/dns/04-cname.txt.
Query the cache twice, a few seconds apart, for the nonexistent name nope.lab.internal with +comments +authority and save the output to /root/dns/05-nx.txt. Then write the number of seconds a negative response stays in the cache, as a single number, in /root/dns/05-negttl.txt.
Query the cache once for www to fill it, then change www.lab.internal. in the zone file to 10.0.0.11 and increase the SOA serial. Then, within 120 seconds, query the authoritative server (5300) and the cache (5301) in turn and save both answers to /root/dns/06-stale.txt.
Stop the cache server, start it again, and save the answer for www to /root/dns/07-after.txt. If you could not flush the cache, write the maximum number of seconds the old answer could remain, as a single number, in /root/dns/07-window.txt.
Query the short name with dig @127.0.0.1 -p 5300 +search +domain=lab.internal +ndots=5 api.lab A, then extract the QUERY lines containing api.lab from the authoritative server log (/root/dns/auth.log) and save them to /root/dns/08-search.txt.
Summarize in /root/dns/09-report.md. It must cover why /etc/hosts comes first, the worst-case propagation time for a migration (in seconds), the negative caching time (in seconds), and the extra queries that ndots creates.

Notes

+noall +answer in dig shows only the answer section, and +comments +authority shows the status line and the authority section.
The authoritative server rereads the zone file on every query. You do not need to restart it after editing. The cache server must be restarted to be emptied.
The "cache gives the old answer" in step 6 is visible only if you observe it within the TTL (120 seconds) after filling the cache. If time has passed, fill the cache again and redo it.
Common mistake: if you query the authoritative server (5300) twice instead of the cache (5301), the TTL does not decrease. The authoritative server always gives the zone's TTL as is.

Zone file and authoritative server

Create /root/dns/zone.txt, start the authoritative server on port 5300, and save the answer for www.lab.internal to /root/dns/01-auth.txt.

A zone file has one record per line in the form 이름 TTL 타입 값 (name, TTL, type, value, in that order). Write names as absolute names with a trailing dot. An SOA value has seven items, 주서버 관리자 serial refresh retry expire minimum (primary server, administrator, serial, refresh, retry, expire, minimum), and the last one, minimum, is the negative-response caching time. Run the server in the background with nohup ... > /root/dns/auth.log 2>&1 &.

/etc/hosts comes first

Add 10.0.0.99 www.lab.internal to /etc/hosts, and save the output of getent hosts www.lab.internal together with the hosts: line of nsswitch to /root/dns/02-hosts.txt.

getent resolves names through the same path (nsswitch) as programs do. If files comes before dns on the hosts: line, /etc/hosts wins. You can extract just that line with grep '^hosts:' /etc/nsswitch.conf.

The cache answers with the TTL counting down

Start the cache server on port 5301, query www.lab.internal twice a few seconds apart, and save both answers to /root/dns/03-ttl.txt.

Cache mode is minidns.py cache --upstream 127.0.0.1:5300 --port 5301. The first query goes to the upstream and is stored; from the second query on, the cache returns the answer with the remaining TTL reduced. Append the second result with >>.

A CNAME chain arrives in one answer

Ask the cache for the A record of api.lab.internal and save the answer to /root/dns/04-cname.txt.

A resolver that meets a CNAME follows the chain to the final A and puts both records in one answer. If you see only one line, you queried for the CNAME type — query for A.

Nonexistent names are remembered too

Query the cache twice, a few seconds apart, for nope.lab.internal with +comments +authority and save the output to /root/dns/05-nx.txt, then write the number of seconds a negative response stays in the cache to /root/dns/05-negttl.txt.

An NXDOMAIN answer has no answer section, and the SOA arrives in the authority section. That SOA's TTL is the caching time of this negative response, and per RFC 2308 it is the smaller of the SOA minimum and the SOA's own TTL. Query twice and keep the output so that you can see that TTL decreasing.

The zone changed but the cache has the old answer

Query the cache for www to fill it, then change www.lab.internal. in the zone to 10.0.0.11 and increase the serial. Within 120 seconds, query the authoritative server and the cache in turn and save both answers to /root/dns/06-stale.txt.

The authoritative server reads the file immediately, so it gives the new address, while the cache gives its stored old answer until the TTL runs out. The order matters — fill the cache first, edit, and then query both right away. You can raise the serial by changing a single digit with sed.

If you flush the cache, and if you cannot

Stop the cache server and start it again, then write the answer for www to /root/dns/07-after.txt, and write the maximum number of seconds the old answer can remain when you cannot flush the cache to /root/dns/07-window.txt.

This cache stores data only in memory, so stopping the process empties it (pkill -f 'minidns.py cache'). You cannot flush someone else's cache (a customer's ISP), and the time you then have to wait is the TTL of the record before the change.

The extra queries that ndots creates

Query with +search +domain=lab.internal +ndots=5 api.lab, then save the QUERY lines containing api.lab from the authoritative server log to /root/dns/08-search.txt.

A name with fewer dots than ndots gets the search domain appended first. So in the log, the api.lab.lab.internal. query comes first, and after it fails, the api.lab. query follows. Extract them with grep 'QUERY api\.lab' /root/dns/auth.log.

Summary report

In /root/dns/09-report.md, summarize the /etc/hosts priority, the worst-case propagation time (in seconds), the negative caching time (in seconds), and the extra queries that ndots creates.

Use the numbers you measured yourself in this lab. The propagation time comes from step 7, and the negative caching time from step 5. For ndots, write down the order of queries you saw in the log in step 8.