Tell apart what is recorded from what is not

This lab runs in a VM

It is a single Ubuntu 24.04 VM. You are root and have kernel privileges, so you can actually load audit rules. The lab Pod has no capabilities at all, so auditctl does not work there; that is why only this lab runs in a VM. The first boot takes 1–2 minutes.

Goal

Put in kernel audit rules yourself, cause an event on purpose, and pull out and read the record. Find the same event again on the journal side to see what each of the two records has, and finally find a place that is not watched and cover it.

Why it matters

There is a question you must answer before you write a detection rule — is that event recorded anywhere in the first place? On Linux, kernel audit records almost nothing by default. It looks only at the paths and syscalls you have put rules on. That is how the situation "we are collecting all the logs, yet there is no answer" arises.

What you learn here is not commands but a sense of scope. If you can say what your rules see and what they do not see, then when an incident happens you can tell "there is no record" apart from "we were not looking at that place." The two lead to completely different remedies.

Steps

1. Save the output of auditctl -s and the service status to /root/det/01-status.txt.
2. Write a rule that watches /etc/cron.d for writes and attribute changes with the key cron_change into /etc/audit/rules.d/labhub.rules and load it into the kernel.
3. Add a rule that catches only 64-bit execve calls whose executable is /usr/bin/curl, with the key outbound_exec, to the same file and load it.
4. Create /etc/cron.d/nightly-report so that it calls curl, run curl by hand once, and write what you did to /root/det/04-trigger.txt.
5. Save the last event of ausearch -k outbound_exec -i to /root/det/05-evidence.txt and write the three values pid=, exe=, and key= beneath it.
6. Extract cron's execution records from the journal as JSON and save them to /root/det/06-cron.json. It must include the execution line of the job that runs every minute on this VM (labhub-metrics).
7. Create /usr/local/bin/labhub-helper.sh and give it the execute bit, confirm that no record of it is left under the cron_change key, and write the command, the result, and the reason to /root/det/07-blindspot.txt.
8. Add a rule that watches /usr/local/bin with the key persistence and load it, modify the file once more, and confirm in /root/det/08-closed.txt that a record is left.
9. Write this host's detection scope to /root/det/09-report.md in four sections, ## 무엇이 남는가, ## 무엇이 남지 않는가, ## 규칙 목록, and ## 다음에 할 일 (in order: what is recorded, what is not recorded, the rule list, and what to do next).

Notes

• If you put rules into auditctl by hand, they disappear on reboot. Write them in /etc/audit/rules.d/ and load them with augenrules --load. You see what has been loaded with auditctl -l.
• Without -i, ausearch shows the uid and syscalls as numbers. Add -i when reading.
• To extract the journal in a machine-readable form, use -o json as in journalctl -t CRON -o json --since "-30 min".
• Why the keys are separate in step 7: if you use the same key in step 8, the observation from step 7 gets overturned. You cover the new place with a new key.
• Common mistake: creating the file before you load the rules. Audit sees only what happens after the rules are loaded.

What is turned on

Save the output of auditctl -s and the result of systemctl is-active auditd to /root/det/01-status.txt. The values of enabled and backlog_limit must be in it as they are.

auditctl -s prints the current state of the kernel audit subsystem, one item per line. Just append the output of the two commands into one file.

Watch the scheduled-job directory

Write a rule that watches /etc/cron.d for writes (w) and attribute changes (a) and attaches the key cron_change into /etc/audit/rules.d/labhub.rules, and load it into the kernel.

A file watch rule has the shape -w <경로> -p <권한> -k <키> (path, permissions, key). The command that loads what you wrote in rules.d into the kernel is augenrules --load, and you check the loaded rules with auditctl -l.

What was executed

Add a rule that catches only 64-bit execve calls whose executable is /usr/bin/curl, with the key outbound_exec, to /etc/audit/rules.d/labhub.rules and load it into the kernel.

A syscall rule has the shape -a always,exit -F arch=b64 -S <시스콜> -F <필터> -k <키> (syscall, filter, key). The filter name that narrows by executable path is exe. A file watch cannot show "what was executed."

Cause the event

Create /etc/cron.d/nightly-report so that it calls curl at three in the morning, and then run curl by hand once. Then write what you did, in two or more lines, to /root/det/04-trigger.txt.

You have to create the file after loading the rules for a record to be left. curl does not actually need to reach anywhere — the fact that it was executed is left as an execve.

Pull out and read the record

Save the last event of ausearch -k outbound_exec -i to /root/det/05-evidence.txt, and beneath it write the three values pid=, exe=, and key=, each on its own line.

Without -i, the uid and syscall number come out as numbers. To see only the last event, you can keep only what comes after the last ---- separator of the ausearch result with awk/sed, or cut it with tail.

The same event in the journal

Extract the execution records cron left in the journal as JSON and save them to /root/det/06-cron.json. It must include the CMD line of the job that runs every minute on this VM (labhub-metrics).

With journalctl -t CRON -o json --since "-30 min" you get one JSON object per line. If it has not run yet, wait a minute and extract again.

A place that is not watched

Create /usr/local/bin/labhub-helper.sh and give it the execute bit, confirm that the path does not appear in ausearch -k cron_change, and write the command, the result, and why it is not recorded, in at least 120 characters, to /root/det/07-blindspot.txt.

Audit rules look only at the paths written in them. An attacker does not need to create a new scheduled job; changing the contents of a script that is already scheduled is enough. This step is where you confirm for yourself that that place is currently outside the watch.

Cover the empty spot

Add a rule that watches /usr/local/bin for writes and attribute changes and attaches the key persistence to /etc/audit/rules.d/labhub.rules and load it, then modify labhub-helper.sh once more and confirm in /root/det/08-closed.txt that a record is left.

If you use the key cron_change, the "nothing is left" you confirmed in step 7 gets overturned and the earlier step breaks. You cover the new place with a new key. You have to modify the file after loading the rules for a record to be left.

Put the detection scope in a document

Write what is and is not recorded on this host right now to /root/det/09-report.md in four sections, ## 무엇이 남는가, ## 무엇이 남지 않는가, ## 규칙 목록, and ## 다음에 할 일 (in order: what is recorded, what is not recorded, the rule list, and what to do next). The three key names and the rules file path must appear in the body.

In the rule list section you can paste the result of auditctl -l. "What is not recorded" is the most valuable section in this document — it is the place that keeps the next person from believing "everything is recorded."