The customer said logs could not leave the building

Goal

You build a support bundle collector that the customer's owner will run directly on the customer server. It gathers by an allowlist, redacts secrets, has a manifest and a size cap, and bundles into a tar.gz that gives the same bytes for the same input.

Why it matters

To receive a bundle from a customer who cannot send logs outside, the customer's security owner has to approve it. If you cannot show what went in, how many redactions were made, and whether the file received is the very file that was reviewed, the bundle cannot go out. If a person picks the files each time, secrets leak or a useless bundle comes out.

Materials: /opt/lab/p1a-bundle/site — a copy of the customer server (VERSION, etc, logs, run/environ, data). /opt/lab/p1a-bundle/rules.tsv — the additional redaction rules of the customer's security team. The working directory is /root/bundle. The expected time is 60 minutes, and when the session ends, /root/bundle disappears.

Collection rules (the promise of this lab):

• What goes in: VERSION, regular files under etc/ (excluding symbolic links), *.log directly under logs/, and run/environ (in stage it is env.txt, with NUL turned into newlines and sorted by name).
• What is redacted: the whole private key block → [REDACTED:private_key], the value after Bearer → [REDACTED:token], values whose key name ends with password, passwd, or secret → [REDACTED:password], values ending with token or api_key → [REDACTED:token], and emails → [REDACTED:email]. The key and the value are separated by : or =. The key name and the separator are kept.
• Log cap: 65536 bytes per file. After redacting, keep only the intact lines from the recent side.

Steps

1. From the material site, write the relative paths of the files that fit the collection rules into /root/bundle/scope.txt, one per line.
2. Create /root/bundle/redact.py, which redacts standard input to standard output and prints redactions=N to standard error.
3. Create /root/bundle/collect.py, where python3 collect.py SRC OUT puts the allowlisted files, redacted, in OUT/stage.
4. Make collect.py create OUT/stage/manifest.json (files array: path, size, sha256, redactions, truncated, sorted by path, plus total_redactions).
5. Apply the 65536-byte cap to the log files and make the manifest's truncated and redactions match reality.
6. Make collect.py create OUT/support-bundle.tar.gz. The entries are in name order under support-bundle/, and for the same input the sha256 must be the same no matter when or where you bundle.
7. Accept --rules FILE, apply kind<TAB>정규식 rules (the placeholder is the regular expression) after the default rules, and redact with [REDACTED:kind].
8. With the material site and rules.tsv, create /root/bundle/delivery/support-bundle.tar.gz, SHA256SUMS, and REDACTION-REPORT.tsv.

Notes

• The grader runs your redact.py and collect.py directly on a server tree where it has planted new secrets each time. A rule fitted to the material values, or hard-coding, fails.
• Over-redaction is also a failure: lines like password_min_length: 12, token_ttl_sec: 3600, passwordless_login: false, and orders-api@sha256:… must remain as they are.
• From step 3 on, you keep growing the single file collect.py. If you delete an earlier step's feature, the earlier step's grading fails again.
• Looking at a NUL-delimited file: tr '\0' '\n' < /opt/lab/p1a-bundle/site/run/environ
• Checking determinism: run the same command twice (changing the source mtime with touch in between) and compare sha256sum.
• Common mistakes: using tarfile.open(..., "w:gz") as it is, putting a generation time in the manifest, and redacting after truncating.

Decide what goes in with an allowlist

From the material site, write the relative paths of the files that fit the collection rules into /root/bundle/scope.txt, one per line.

First scan the whole site with find. Include only what the rules name, and rotated logs, logs in subdirectories, customer data, and pid files are left out because they are not in the rules. The paths are relative to site.

A narrow and precise redaction filter

Create /root/bundle/redact.py, which redacts standard input to standard output and prints redactions=N to standard error.

Define the regular expressions by shape. A private key is multi-line, so replace it as a whole block with re.S before the line rules. The key name must 'end with' password, not just 'contain' it, for password_min_length to survive. A value may be wrapped in quotes, or may end at & or whitespace.

Gather only the allowlist and build a redacted stage

Create /root/bundle/collect.py, where python3 /root/bundle/collect.py SRC OUT puts the allowlisted files, redacted, in OUT/stage.

If you import redact.py from the same directory, you do not have to write the rules twice. When walking under etc, filter out links with os.path.islink, and split run/environ on NUL, sort it, and put it in env.txt. Make sure that when you run again, no files from the previous stage remain.

A manifest for the reviewer to reconcile

Make collect.py create OUT/stage/manifest.json (files: path, size, sha256, redactions, truncated, in path order, plus total_redactions).

Compute sha256 and size from the bytes actually written to stage after redacting. redactions is the number of redactions in that file. If you put a generation time in this file, think ahead about what will become a problem in step 6.

For large logs, redact and then keep only the recent lines

Apply a 65536-byte cap to each file in logs, and make the manifest's truncated and redactions match the truncated result.

After taking the last 65536 bytes of the redacted result, discard the first line of that window if it is not intact. You can tell by whether the byte right before the window is a newline. The count has to be counted again on the remaining part, excluding the part that was cut off.

The same bytes for the same input

Make collect.py create OUT/support-bundle.tar.gz. The entries are in name order under support-bundle/, and the sha256 must be the same even if you bundle at a different time or to a different OUT.

tar holds the modification time, owner, permissions, and order, and the gzip header holds the time once more. In Python, build the TarInfo yourself to fix the values and set the mtime of GzipFile. If you bundle with GNU tar, look at the options in the reproducible-builds documentation.

Receive the customer security team's additional rules as a file

Make collect.py accept --rules FILE (kindregular expression), apply it after the default rules, and merge the counts into the manifest.

Skip comments (#) and empty lines. Do not hard-code the rules; read them from the file — the grader gives different kind names and regular expressions each time, and also checks whether those values remain when you run without the rules.

The delivery bundle and the report for review

With the material site and rules.tsv, create /root/bundle/delivery/support-bundle.tar.gz, /root/bundle/delivery/SHA256SUMS, and /root/bundle/delivery/REDACTION-REPORT.tsv.

SHA256SUMS is in a format that the receiving side can check with sha256sum -c. For the counts in the report, do not copy the manifest; count the markers by kind in the files inside the bundle and write those. The grader bundles again from the material with the current collect.py and checks whether it is byte-identical to the delivered one.