Writing a systemd Unit File

Goal

You write a systemd unit file from scratch, make a timer pair, and build a unit verification script yourself.

Why it matters

The most important line in a unit file is Type=. If it is Type=simple but the process daemonizes, systemd sees the parent dying as the service ending and treats it as a failure. Conversely, if it is Type=forking but the process runs in the foreground, systemd waits forever and then kills it on a timeout — the typical cause of "the start hangs for 30 seconds and then fails".

And without an [Install] section, systemctl enable does nothing, so automatic start at boot does not happen. Nobody knows this until a reboot.

This lab runs on an AlmaLinux 9 virtual machine. Because systemd really runs as PID 1, after writing a unit file you can start it right away with systemctl enable --now. You learn by seeing with your own eyes what action a single line of the file leads to.\n\nIt used to run in a Pod, where systemctl itself did not exist and writing and verification were all there was. It takes about 30 seconds to start.

Steps

1. Create /etc/systemd/system/labhub-api.service and put in the three sections [Unit], [Service], and [Install]. Also create the /root/unit working directory.
2. In [Unit], put four lines: Description, Documentation, After=network-online.target, and Wants=network-online.target.
3. In [Service], put Type=notify, ExecStart=/usr/local/bin/labhub-api --config /etc/labhub/api.yaml, User=labhub, Group=labhub, and WorkingDirectory=/opt/labhub. ExecStart must be an absolute path.\n\n Because the lab program is a shell script, also put in NotifyAccess=all. The default main accepts only notifications sent by the MainPID, but systemd-notify is a child of the script, so its PID differs. Then systemd cannot receive the readiness signal and waits until it ends with Job for labhub-api.service failed because a timeout was exceeded — there seems to be nothing wrong with the unit file.
4. Add a restart policy. Put Restart=on-failure and RestartSec=5 in [Service], and StartLimitIntervalSec=300 and StartLimitBurst=5 in [Unit]. These two moved from [Service] to [Unit] in systemd 230 — if you write them in [Service], only Unknown key name is left and they are silently ignored, so no rate limit applies.
5. Add the security directives. NoNewPrivileges=true, ProtectSystem=strict, ProtectHome=true, PrivateTmp=true, ReadWritePaths=/var/lib/labhub /var/log/labhub
6. Put WantedBy=multi-user.target in [Install], and create by hand the symlink that systemctl enable would create. The path is /etc/systemd/system/multi-user.target.wants/labhub-api.service and it must point to the original unit.
7. Create a timer pair. /etc/systemd/system/labhub-backup.service (Type=oneshot, an absolute path for ExecStart) and /etc/systemd/system/labhub-backup.timer (in [Timer], OnCalendar=*-*-* 02:30:00, Persistent=true, and Unit=labhub-backup.service, and in [Install], WantedBy=timers.target).
8. Write /root/unit/lint.sh. It must take the unit file path as its first argument, check the following three rules, and end with exit code 0 if there are no violations and a nonzero value if there are.
   • Rule 1: all three sections [Unit], [Service], and [Install] must be present
   • Rule 2: the ExecStart value must be an absolute path (starting with /)
   • Rule 3: if Type=forking, PIDFile= must be present The grader runs it against both your labhub-api.service (it must pass) and /opt/fixtures/unit/bad-sample.service (it must fail).

Notes

• You create the symlink with ln -sf /etc/systemd/system/labhub-api.service /etc/systemd/system/multi-user.target.wants/labhub-api.service. Create the directory first.
• Shell features (pipes, redirection, wildcards) do not work in ExecStart. If you need them, wrap it in /bin/bash -c '...'.
• This lab runs on an AlmaLinux 9 VM and systemd is PID 1. So you can start the unit you wrote right away with systemctl start, and step 8 is actually graded that way. It used to run in a Pod, where systemctl itself did not exist.
• You may create the symlink in step 6 by hand, but check once that the result is the same as what systemctl enable creates. Later, when you meet "I enabled it, so why doesn't it start", you will look at the symlink first.
• Common mistake 1: the script in step 8 matches section headers as substrings and cannot tell [Unit] from [Unite].
• Common mistake 2: copying the file instead of making a symlink in step 6. systemd manages it with symlinks.

Three-section skeleton

Create /etc/systemd/system/labhub-api.service and put in the three sections [Unit], [Service], and [Install]. Also create the /root/unit working directory.

You need three section names wrapped in square brackets. There is a convention for the order too.

Filling in the [Unit] section

In [Unit], put four lines: Description, Documentation, After=network-online.target, and Wants=network-online.target.

You write four things: description, documentation, ordering, and dependency. Ordering and dependency are different directives.

[Service] execution settings

In [Service], put Type=notify, ExecStart=/usr/local/bin/labhub-api --config /etc/labhub/api.yaml, User=labhub, Group=labhub, and WorkingDirectory=/opt/labhub. ExecStart must be an absolute path.\n\n Because the lab program is a shell script, also put in NotifyAccess=all. The default main accepts only notifications sent by the MainPID, but systemd-notify is a child of the script, so its PID differs. Then systemd cannot receive the readiness signal and waits until it ends with Job for labhub-api.service failed because a timeout was exceeded — there seems to be nothing wrong with the unit file.

The type and the execution command are the key. The execution path must be an absolute path.

Restart policy

Add a restart policy. Put Restart=on-failure and RestartSec=5 in [Service], and StartLimitIntervalSec=300 and StartLimitBurst=5 in [Unit]. These two moved from [Service] to [Unit] in systemd 230 — if you write them in [Service], only Unknown key name is left and they are silently ignored, so no rate limit applies.

There are two limit directives that prevent an infinite loop. The interval must not be 0 either.

Security hardening

Add the security directives. NoNewPrivileges=true, ProtectSystem=strict, ProtectHome=true, PrivateTmp=true, ReadWritePaths=/var/lib/labhub /var/log/labhub

Use together the directive that makes things read-only and the directive that specifies exception paths.

[Install] and the result of enable

Put WantedBy=multi-user.target in [Install], and create by hand the symlink that systemctl enable would create. The path is /etc/systemd/system/multi-user.target.wants/labhub-api.service and it must point to the original unit.

If you create by hand the symlink path that enable creates, the structure becomes clear.

Writing a timer pair

Create a timer pair. /etc/systemd/system/labhub-backup.service (Type=oneshot, an absolute path for ExecStart) and /etc/systemd/system/labhub-backup.timer (in [Timer], OnCalendar=*-*-* 02:30:00, Persistent=true, and Unit=labhub-backup.service, and in [Install], WantedBy=timers.target).

A .timer and a .service are a pair. Also put in the directive that catches up on missed runs.

Unit verification script

Write /root/unit/lint.sh. It must take the unit file path as its first argument, check the following three rules, and end with exit code 0 if there are no violations and a nonzero value if there are.

• Rule 1: all three sections [Unit], [Service], and [Install] must be present
• Rule 2: the ExecStart value must be an absolute path (starting with /)
• Rule 3: if Type=forking, PIDFile= must be present The grader runs it against both your labhub-api.service (it must pass) and /opt/fixtures/unit/bad-sample.service (it must fail).

The grader runs it against both a good unit and the fixture's bad unit. Check the three rules.