Names and time in an air-gapped network

In one line

An air-gapped network needs names and time too. The internal DNS must answer for the internal zone authoritatively, without an upstream, so that unknown names do not wait for a timeout, and the internal time server must declare itself the reference even with no upstream, so that the clocks of the servers do not scatter from one another.

Why this was needed

To call the internal mirror pypi.airgap.internal, there must be a DNS that resolves that name. Editing /etc/hosts on every server goes wrong once you have more than about ten servers. But if you stand up the internal DNS as usual, it forwards unknown names upstream, and since the upstream of an air-gapped network does not answer, the client waits for a timeout on every query. Time breaks more quietly. A server that cannot reach internet time servers drifts on its own clock, and in a month the servers spread apart from each other by tens of seconds to minutes. Then certificates are rejected as "not yet valid", and the logs of several servers can no longer be lined up by timestamp.

How it works

The internal zone with dnsmasq. According to the dnsmasq manual, listen-address and bind-interfaces open sockets only on the specified addresses (Ubuntu's systemd-resolved stub uses port 53 on 127.0.0.53 and 127.0.0.54, so without these they collide). no-resolv stops it from reading upstreams from resolv.conf, and local=/airgap.internal/ makes it not forward queries of that zone outside and answer only from /etc/hosts and DHCP (--local is another name for --server). domain= sets the zone to attach to short names, and addn-hosts= sets an additional name-list file to read besides /etc/hosts. address=/도메인/주소 (the placeholders are the domain and the address) answers even subnames with a single address, but unless you want to tie the whole internal zone to one address, a name-list file is easier to manage.

Applying changes without a restart. When dnsmasq receives SIGHUP, it clears its cache and rereads /etc/hosts and the addn-hosts files. It does not reread the configuration file itself. So it is more convenient in operation to keep frequently changing name lists apart from the configuration.

The client side. ip netns exec mounts /etc/netns/<이름>/resolv.conf (the placeholder is the namespace name), if it exists, in the place of /etc/resolv.conf inside that namespace. On a real server, this corresponds to writing the internal DNS and a search domain in /etc/resolv.conf (or in systemd-resolved's DNS settings).

Internal time with chrony. The chrony documentation says that by default it accepts no clients and runs as a pure NTP client. You need allow <대역> (the placeholder is the address range) for it to serve time. local stratum N (1 to 15, default 10) makes it serve its own clock as the reference of that stratum even without upstream synchronization. To have several machines take each other as reference, you use the orphan option. To check from a client without touching the clock, chronyd -Q prints only the difference and exits (-q adjusts once and exits), and on the server, chronyc tracking and chronyc clients show the state and the queries received.

The limit of a single reference clock. local stratum is only a declaration that "this clock is right"; it does not make it right. The whole air-gapped network follows one clock, so they do not drift apart from each other, but if that clock is wrong, all are wrong together. In the field, you install a hardware reference such as a GPS or standard-time receiver, or set up a procedure of setting the time, at each import, from a time a person has checked.

What it looks like in the field

I measured this in this lab VM. With only no-resolv and without local=, both nope.airgap.internal, which is not in the name list, and the outside name ubuntu.com returned status: REFUSED. With no upstream to forward to, it means "I will not answer here". When I added local=/airgap.internal/, the same internal name changed to NXDOMAIN (an authoritative "does not exist"). A client that receives REFUSED looks for the next name server, and a client that receives NXDOMAIN stops there — which is why, at least for the internal zone, you must give authoritative answers. For chrony, when I added local stratum 8 and restarted it, chronyc tracking showed Reference ID : 7F7F0101 () and Stratum : 8, and systemctl reload dnsmasq was kill -HUP $MAINPID, as written in the unit.

The VM in this lab cannot reach outside NTP (UDP 123) from the start — only public 80/443 and DNS are open. So right after installation, chrony is running without having synchronized to a single upstream server. That is exactly the sight you often see on air-gapped servers. If you block outbound traffic on top of that, name resolution is cut as well.

What you will do in the next lab

You block the VM's outbound traffic to make a real air-gapped network, and stand up dnsmasq as an authoritative server for airgap.internal with no upstream. You make the client namespace use that DNS and the search domain, and add a name without a restart. You turn chrony into the internal time server, query it from client, and finish with a handover check script.

Reference documents: dnsmasq(8) · resolved.conf(5) · ip-netns(8) · chrony.conf(5) · chronyd(8) · chronyc(1)