Running now is not the same as starting at boot

In one line

"It is up" and "it comes up at boot" are different facts. The former means a process exists now, and the latter means that enable has made a symlink so that systemd puts that service into the boot transaction. A handoff of a customer VM is finished only when you prove the latter.

Why this was needed

The most common installation in the field goes like this. You copy the app, type nohup python3 app.py & in a shell, open it in a browser, and report that "it works well". A few weeks later the customer reboots the VM for a security patch, and nothing comes up. That process was a process the service manager did not know about. If it dies, nobody revives it, and at boot nobody starts it. The log is somewhere in nohup.out, and nobody knows who deleted even that file.

If you put it up as a systemd service, these questions get answers. Who starts it (systemd), as which user (User=), where the settings come from (EnvironmentFile=), what happens when it dies (Restart=), when it stands at boot (WantedBy= and After=), and where the logs are (the journal). If an FDE installs anything on a customer VM, these six lines are the backbone of the handoff document.

How it works

A unit file and the loaded configuration are different. A unit made by an administrator is placed in /etc/systemd/system/, and this path takes precedence over the distribution package's /usr/lib/systemd/system/. But systemd does not read the file every time. It runs on the configuration it has loaded, so if you fixed the file of an already loaded unit, it has to be reread with systemctl daemon-reload. If you forget the reload after fixing, it shows up as NeedDaemonReload=yes in systemctl show, and systemctl gives a warning.

Wants and After do not substitute for each other. The systemd.unit manual states that requirement dependencies (Wants=, Requires=) do not affect the start order, and that the order is set separately with After= and Before=. So for a service that has to come up after the network is configured, as the systemd.special manual says, you pull in network-online.target with Wants= and at the same time stand after it with After=. If you write only one side, it only pulls it in with no order, or there is only an order and nobody pulls in that target.

enable is not a start but a symlink. The [Install] section is not interpreted during normal running. systemctl enable reads WantedBy= and creates a symlink in multi-user.target.wants/, and that symlink acts as the Wants= dependency from multi-user.target to this service at boot. Conversely, systemctl start only starts it once now and does not create a symlink. So "it is up but does not come up after a reboot" is almost always a missing enable.

The default of Restart= is no. on-failure restarts on a nonzero exit code, termination by a signal, a timeout, and the watchdog. However, the manual counts SIGHUP, SIGINT, SIGTERM, and SIGPIPE as a clean exit. So if you kill it with kill -9 (SIGKILL), it comes back, but neither systemctl stop nor an ordinary kill (SIGTERM) triggers on-failure. The default of the restart interval RestartSec= is 100ms, and if it starts too often, it hits the limits of StartLimitIntervalSec= and StartLimitBurst= and does not start any more (you clear it with reset-failed).

EnvironmentFile is read when the process is started. The manual says these files are read right before the process is executed. Each line is KEY=VALUE, lines that start with # are ignored, and if the file does not exist, the service fails to start (if you put - in front of the path, it goes on even without it). So if you changed a token, the unit file stays the same, so no daemon-reload is needed, and you have to restart for the new process to receive the new value.

Sandboxing is seen as a score. You confine a service with the options of systemd.exec. NoNewPrivileges=yes keeps it from gaining new privileges through execve, ProtectSystem=strict mounts the whole filesystem except /dev, /proc, and /sys as read-only (you open the paths to write with ReadWritePaths=), ProtectHome=yes makes /home, /root, and /run/user look empty, and PrivateTmp=yes gives it a private /tmp. systemd-analyze security <유닛> (the placeholder is the unit) scans these settings and produces an exposure level between 0.0 and 10.0. The higher it is, the less it is confined. As the manual emphasizes, this score looks only at the devices that systemd has applied, so it is not a verdict of "vulnerable" but a yardstick for comparison.

[Unit]
Wants=network-online.target
After=network-online.target

[Service]
User=orders
EnvironmentFile=/etc/orders/orders.env
ExecStart=/usr/bin/python3 /opt/orders-svc/app.py
Restart=on-failure
ProtectSystem=strict
ReadWritePaths=/var/lib/orders

[Install]
WantedBy=multi-user.target

What it looks like in the field

If you follow the previous owner's process on the lab VM (measured), /proc/<pid>/cgroup comes out as /system.slice/cloud-final.service. It is merely riding inside the unit that ran the preparation script, and that unit has no responsibility to start this app again. Even if you put up a new service, the first start fails, because the stray process is still holding the port. systemctl start is Type=simple, so it returns as a success the moment it starts the process, and the cause is visible only in the journal's bind failed … Address already in use. With Restart=on-failure and RestartSec=1, it repeats the same failure every second, and Failed with result 'exit-code' piles up line after line in the journal (measured). If it goes past 5 times in this state, it hits the default start limit (5 times in 10 seconds). Conversely, kill -9 came back with a new PID 1 second after status=9/KILL, and an ordinary kill (SIGTERM) ended with Deactivated successfully and stayed inactive (measured).

There is also a typical order when you turn on sandboxing. If you fix the file, systemctl warns "Warning: The unit file, source configuration file or drop-ins of orders.service changed on disk. Run 'systemctl daemon-reload' to reload units." (measured). If you reload and restart, and you left out ReadWritePaths, the app cannot write to the data directory and dies right away. The service in this lab had an exposure score of 9.2 with only User=orders given, and it went down to 8.3 when the five options above were added (measured).

What really matters in practice

• Report an installation with the output of systemctl is-enabled and systemctl show -p Wants,After,Restart,User. "It opens in the browser" is only evidence that it is up now.
• In a place where you cannot reboot (a customer production VM, remote grading), prove boot startup with the enable state, the symlink, and the target dependency. If you make it a check script and put it in the handoff package, the next owner asks the same question in the same way.
• If you fix a unit file, daemon-reload; if you fix an environment file, restart. If you mix the two up, it becomes "I changed it but it did not change".
• Test restarts with kill -9, but remember that SIGTERM is not revived by on-failure.
• An environment file that holds a token is root:the service group 640. A token that has once been written on a wiki is already a leaked value.

What you will do in the next lab

On an Ubuntu VM, you find the app the previous owner started with nohup and record what it is, then create a dedicated account and an environment file and put up the unit. You find the cause of the first start failure in the journal, take down the stray process and stand up the service, and check the symlink that enable created. You see whether it comes back with kill -9, add sandboxing and record the daemon-reload warning and the change in exposure score, and replace the token and apply it with a restart. Finally, you run a check script that judges the boot conditions without a reboot on the prepared units.

References: systemd.service(5) · systemd.exec(5) · systemd.unit(5) · systemd.special(7) · systemd-analyze(1)