LFCS — Linux Foundation System Administrator

From systemd Timers to podman — Build Services and Networking for Real

Goal

You actually run, on an Ubuntu 24.04 VM, what Pods had blocked in the LFCS Operations and Networking domains: systemd services and timers, an nginx reverse proxy, iptables DNAT/DROP, a bridge and a bond on dummy interfaces with netplan, chrony and the time zone, a git merge, a podman image without a registry, a persistent sysctl, and load metrics.

Why it matters

The LFCS looks at "results", not "explanations". Even if you write the unit file well, if systemctl is-enabled says disabled it is 0 points, and even if the nginx configuration is syntactically correct, if the default site is holding 80, the proxy does not work. The grader of this lab looks not only at files but at the state of the system as it is now with systemctl is-active, curl, iptables -S, /sys/class/net, sysctl -n, and podman inspect.

Three principles recur. First, enable and start are different questions (should it come on at boot / is it on now). Second, writing a file and the kernel or daemon knowing about it are different (sysctl --system, daemon-reload, nginx reload). Third, leave a way back — if you change the INPUT default policy of iptables to DROP or block 8899 or 22, the grading agent is cut off and from the next step you get failures of unknown cause.

A session starts at 60 minutes and can be extended up to 180 minutes, and the VM disappears when the session ends. This lab is set at 85 minutes, so extend it once as you start.

Steps

Make /usr/local/bin/lfcs-heartbeat.sh append the current time as one line to /var/log/lfcs-heartbeat.log, and write it into /etc/systemd/system/ as a Type=oneshot lfcs-heartbeat.service and a lfcs-heartbeat.timer that calls that service every minute. The timer must be both enabled and started, and you must start the service once directly so that the log has at least one line.
Put the phrase backend ok in /srv/backend/index.html, and make python3 -m http.server 8000 --directory /srv/backend into a systemd service lfcs-backend.service and enable and start it. Then create /etc/nginx/sites-available/lfcs-proxy so that nginx reverse-proxies to this backend on port 80, and activate it (disable the default site). curl http://127.0.0.1/ must return backend ok.
Add two rules with iptables. (1) In the nat table PREROUTING, a DNAT (or REDIRECT) that sends what comes in on tcp 8080 to 80. (2) In the filter table INPUT, DROP tcp 9999 on all interfaces. Leave the INPUT default policy as ACCEPT, and do not touch 8899 and 22 (the grading agent and SSH). Finally, save the iptables-save output to /root/lfcs/net/rules.v4.
Create dummy interfaces dummy0, dummy1, and dummy2. Put dummy0 in a bridge br0 and give br0 192.0.2.1/24. Create a bond bond0 in active-backup mode and put dummy1 and dummy2 in it (do not touch real interfaces such as eth0). Write the same configuration in netplan syntax in /root/lfcs/net/99-lab.yaml, but do not apply it.
Add the line server time.cloudflare.com iburst to /etc/chrony/chrony.conf and restart chrony. Change the time zone to Asia/Seoul. Then save chronyc sources to /root/lfcs/time/sources.txt, chronyc tracking to /root/lfcs/time/tracking.txt, and timedatectl to /root/lfcs/time/timedatectl.txt. (On this VM UDP 123 is blocked, so synchronization itself may not happen — the configuration and state are what is graded.)
Create a git repository in /root/lfcs/repo (default branch main) and commit README.md. On a branch feature, add feature.txt and commit it, then return to main and merge so that a merge commit remains (--no-ff). The working tree must be clean.
Bundle a root filesystem holding only /bin/busybox (busybox-static) into a tar and make the image localhost/lfcs-busybox:1 with podman import (you do not pull from a registry). Run a container named bb1 from that image with --network none to print hello-from-busybox, and save that output to /root/lfcs/bb/run.txt. Do not delete the container (the exit code must be 0).
In /etc/sysctl.d/90-lfcs.conf, write three values, net.ipv4.ip_forward = 1, vm.swappiness = 10, and net.core.somaxconn = 1024, and apply them to the current kernel as well. After applying, read the three values with sysctl and save them to /root/lfcs/sysctl/runtime.txt.
Save the uptime output to /root/lfcs/perf/uptime.txt and the first 15 lines of top -bn1 to /root/lfcs/perf/top.txt. Then make /root/lfcs/perf/load.txt with two lines: NPROC=<이 VM 의 CPU 개수> and LOAD1=<1분 로드 평균(/proc/loadavg 첫 값)> (NPROC= followed by the number of CPUs of this VM, and LOAD1= followed by the 1-minute load average, the first value of /proc/loadavg).

Notes

After writing or editing a unit, run systemctl daemon-reload. Check whether the timer was picked up with systemctl list-timers --all.
nginx: nginx -t → systemctl reload nginx. If you do not remove sites-enabled/default, there become two default_server.
iptables: iptables -t nat -S PREROUTING, iptables -S INPUT. You can ask whether a rule already exists with -C.
For a bond slave, run master bond0 after ip link set dummy1 down. To check the mode, cat /sys/class/net/bond0/bonding/mode.
This VM has only DNS and 80/443 open to the outside. The ? in chronyc sources is normal, and podman does not pull from a registry.
Common mistake 1: only running start on the timer and leaving out enable — it disappears on reboot.
Common mistake 2: like lvextend, writing only the file for sysctl too and leaving out sysctl --system, so the kernel value stays the same.
Common mistake 3: merging with fast-forward so there is no merge commit.

Service unit and timer

Make /usr/local/bin/lfcs-heartbeat.sh append the current time as one line to /var/log/lfcs-heartbeat.log, and write it into /etc/systemd/system/ as a Type=oneshot lfcs-heartbeat.service and a lfcs-heartbeat.timer that calls that service every minute. The timer must be both enabled and started, and you must start the service once directly so that the log has at least one line.

A timer needs OnBootSec/OnUnitActiveSec in [Timer] and WantedBy=timers.target in [Install]. After writing the unit files, daemon-reload comes first. is-enabled and is-active are different questions.

nginx reverse proxy

Put the phrase backend ok in /srv/backend/index.html, and make python3 -m http.server 8000 --directory /srv/backend into a systemd service lfcs-backend.service and enable and start it. Then create /etc/nginx/sites-available/lfcs-proxy so that nginx reverse-proxies to this backend on port 80, and activate it (disable the default site). curl http://127.0.0.1/ must return backend ok.

The key points are listen 80 default_server in the server block and location / { proxy_pass http://127.0.0.1:8000; }. If you do not remove sites-enabled/default, the two default_server conflict. After checking with nginx -t, reload.

DNAT and DROP with iptables

Add two rules with iptables. (1) In the nat table PREROUTING, a DNAT (or REDIRECT) that sends what comes in on tcp 8080 to 80. (2) In the filter table INPUT, DROP tcp 9999 on all interfaces. Leave the INPUT default policy as ACCEPT, and do not touch 8899 and 22 (the grading agent and SSH). Finally, save the iptables-save output to /root/lfcs/net/rules.v4.

DNAT is -t nat -A PREROUTING -p tcp --dport 8080 -j DNAT --to-destination :80 (or -j REDIRECT --to-ports 80). DROP is -A INPUT -p tcp --dport 9999 -j DROP. A packet you send to yourself does not pass through PREROUTING, so you check DNAT by the existence of the rule, and DROP by whether a connection to 127.0.0.1:9999 is a 'timeout' rather than a 'refusal'.

Bridge and bond on dummies, and netplan

Create dummy interfaces dummy0, dummy1, and dummy2. Put dummy0 in a bridge br0 and give br0 192.0.2.1/24. Create a bond bond0 in active-backup mode and put dummy1 and dummy2 in it (do not touch real interfaces such as eth0). Write the same configuration in netplan syntax in /root/lfcs/net/99-lab.yaml, but do not apply it.

ip link add type dummy|bridge|bond. For a slave, ip link set master . A bond slave must be down before it is added. Check the bond mode with /sys/class/net/bond0/bonding/mode.

chrony and time zone

Add the line server time.cloudflare.com iburst to /etc/chrony/chrony.conf and restart chrony. Change the time zone to Asia/Seoul. Then save chronyc sources to /root/lfcs/time/sources.txt, chronyc tracking to /root/lfcs/time/tracking.txt, and timedatectl to /root/lfcs/time/timedatectl.txt. (On this VM UDP 123 is blocked, so synchronization itself may not happen — the configuration and state are what is graded.)

timedatectl set-timezone Asia/Seoul. The name of the chrony service on Ubuntu is chrony. A ? in the first column of chronyc sources means no response has been received yet, and here it is normal.

git basics — branch and merge

Create a git repository in /root/lfcs/repo (default branch main) and commit README.md. On a branch feature, add feature.txt and commit it, then return to main and merge so that a merge commit remains (--no-ff). The working tree must be clean.

git init -b main. To commit you need user.name/user.email (git config). A merge commit shows with git log --merges. If you merge with fast-forward, there is no merge commit.

Build and run a podman image without a registry

Bundle a root filesystem holding only /bin/busybox (busybox-static) into a tar and make the image localhost/lfcs-busybox:1 with podman import (you do not pull from a registry). Run a container named bb1 from that image with --network none to print hello-from-busybox, and save that output to /root/lfcs/bb/run.txt. Do not delete the container (the exit code must be 0).

Copy rootfs/bin/busybox and run tar -C rootfs -cf bb.tar . → podman import bb.tar localhost/lfcs-busybox:1. Run it with podman run --name bb1 --network none localhost/lfcs-busybox:1 /bin/busybox echo hello-from-busybox. Podman in this lab runs as root.

Make kernel parameters persistent

In /etc/sysctl.d/90-lfcs.conf, write three values, net.ipv4.ip_forward = 1, vm.swappiness = 10, and net.core.somaxconn = 1024, and apply them to the current kernel as well. After applying, read the three values with sysctl and save them to /root/lfcs/sysctl/runtime.txt.

sysctl --system reads /etc/sysctl.d/*.conf in order and applies them. If you only write the file, the kernel does not know until a reboot. See the current value with sysctl -n .

Leave load metrics

Save the uptime output to /root/lfcs/perf/uptime.txt and the first 15 lines of top -bn1 to /root/lfcs/perf/top.txt. Then make /root/lfcs/perf/load.txt with two lines: NPROC=<이 VM 의 CPU 개수> and LOAD1=<1분 로드 평균(/proc/loadavg 첫 값)> (NPROC= followed by the number of CPUs of this VM, and LOAD1= followed by the 1-minute load average, the first value of /proc/loadavg).

top -bn1 prints only once in batch mode. The first field of /proc/loadavg is the 1-minute average, and the number of CPUs is nproc. A load average means something only when compared with the number of CPUs.