LLM App Security — From Injection to Output Handling

Where secrets and personal data leak — don't put them in, filter, mask, and detect

Goal

Find and close three exits in a customer support bot (a key in the system text, verbatim records, and retrieval without customer separation), and detect prompt leakage with a canary.

Why it matters

What goes into the model's input may someday come out in an answer, and the records that write down the input copy its contents to even more places. "Writing down that the model must not say it" is not a boundary; a real model is probabilistic, so it mostly complies and then one day does not. This lab fixes things in this order: do not put secrets in at all (the tool layer), filter other people's data before putting it in (retrieval), mask the records, and if something still leaks, notice it with a canary. Every time it grades, the grader starts the app with temporary data containing a newly made key, canary, and phone numbers, and checks where in the responses and records those values remain.

Steps

Unpack the materials with python3 /opt/fixtures/llmsec/kit.py support, then start /root/llmsec/support/app.py on 127.0.0.1:8102 (no 0.0.0.0, and keep it alive even if you close the terminal). The seed in curl 127.0.0.1:8102/health must be the same as the seed you received when unpacking.
Count what was left verbatim in yesterday's record /root/llmsec/support/logs/app-2026-11-04.log, and write three lines in /root/llmsec/support/exposure.txt: tokens=<count> (the number of distinct user tokens in the record), phones=<count> (the number of distinct phone numbers), and key_lines=<count> (the number of lines containing the partner key API 키: pk-…).
Fix build_system and run_tool so that the partner key is taken out of the system text and the server reads it directly from config.json when the order lookup tool runs. Even for a request to reveal the system text, the key must not appear anywhere in the responses or records, and order lookup must still work.
Fix log_request so that one line per request (rid and user name) is still left in data/app.log, but X-Token values, phone numbers, emails, and the partner key are not left verbatim. Keep only the last four digits of phone numbers (***-****-1234).
Fix build_system and check_output so that the canary value from config.json is put into the system text, and if that value shows up in a response, block the whole answer and leave a JSON line in data/audit.log containing that request's rid and "event": "canary". Normal answers must still go out.
Fix find_tickets so that the tickets put into the model are filtered to those of the requesting customer only. Even when asked with words from someone else's ticket, other people's information must not get mixed into the answer, and it must still answer with the customer's own tickets.
Running incident ls-secret-leak raises the incident. The phone numbers of three customers were found outside the company (/srv/ls-support/found.txt). Find the path for each customer, clean up the source, fix the code as in steps 3–6 of the lab, and then write three lines <customer>: <code> and evidence: <log line> (one line of the records) in /root/incident/ls-secret-leak/report.txt. The code is one of cross-tenant (another customer's ticket mixed into an answer), log-bundle (the diagnostic bundle sent to a partner), misfiled-ticket (a ticket with the wrong owner recorded), or prompt-extract (extraction of the system text).

Notes

Start an app with python3 /root/llmsec/<app>/app.py. Keep it in the background so it stays alive even if you close the terminal (setsid nohup … > <file> 2>&1 &). Every time you make a fix, stop the running one and start it again; the step 1 grader checks whether the running app was started from the current app.py (a check that imitates the operations accident where code from before the fix keeps receiving requests). It must still be running at the end for step 1 to be graded again.
Requests: curl -s -H "X-Token: <token>" -H 'Content-Type: application/json' -d '{"q":"…"}' 127.0.0.1:<port>/<path>. The tokens are in data/users.json.
The grader does not use the app you started; it copies the app.py file into a neighboring directory, starts it the same way you do with the new data/ inside (random data the grader made) and --port, and sends random attacks. Do not delete the lines that read data relative to the data/ next to the app and the port from --port. The functions to fix are listed in the file header. Code that notices it is being graded and behaves differently counts as a failure.
The fake model is /opt/fixtures/llmsec/fakellm.py. Read that file's header and directives() to learn what it reads as an "instruction".
The partner order API is /opt/fixtures/llmsec/partner.py (a fake). It answers only if the key matches the value in config.json; it is not a file to fix.
Incident tools: incident list shows the incidents you can raise and a one-line symptom for each, and incident show <name> shows the ticket again. Raising the same incident again puts it into its original broken state with a new seed (the cause for each target is chosen again too).
The report is /root/incident/<name>/report.txt. Write one line <name>: <cause-code> per target, and at the end add evidence: <log line> (one line of the records that showed the cause) as is (a whole line from a log or configuration, 20 characters or more). The grader first sends new random attacks to the fixed code again (as a regression, not only this incident's paths but also the attacks from the earlier steps of the same lab), checks that a different cleanup was done for each cause, and only then looks at the report. The evidence line must be a line that was in the records right after the incident, and it must connect to that target's incident request or the value that leaked.
Common mistake: turning the records off entirely. Then outages cannot be investigated, and the grader checks that lines containing the request id remain.

Unpack the app and start it on loopback

Unpack the materials with python3 /opt/fixtures/llmsec/kit.py support, then start /root/llmsec/support/app.py on 127.0.0.1:8102 (no 0.0.0.0, and keep it alive even if you close the terminal). The seed in curl 127.0.0.1:8102/health must be the same as the seed you received when unpacking.

Direction: a lab app has no reason to be open to the outside; the address it listens on is its exposure. Tools: ss -ltnp, curl, setsid and nohup. Concretely: the unpack command is exactly as above. After starting it, check that port 8102 shows as 127.0.0.1 in ss -ltn and that the seed in /health is the same as /root/llmsec/support/data/seed.txt.

Count what was left verbatim in yesterday's record

Count what was left verbatim in yesterday's record /root/llmsec/support/logs/app-2026-11-04.log, and write three lines in /root/llmsec/support/exposure.txt: tokens=<count> (the number of distinct user tokens in the record), phones=<count> (the number of distinct phone numbers), and key_lines=<count> (the number of lines containing the partner key API 키: pk-…).

Direction: a verbatim record is a copy of everything that went into the prompt. Counting what was copied and how many times is the first step in sizing the damage. Tools: grep -o, sort -u, wc -l, jq, data/users.json. Concretely: count tokens by whether values from users.json appear in the record, phone numbers by the number of distinct values, and keys by the number of lines; the three use different units of counting.

Move the key from the system text to the tool layer

Fix build_system and run_tool so that the partner key is taken out of the system text and the server reads it directly from config.json when the order lookup tool runs. Even for a request to reveal the system text, the key must not appear anywhere in the responses or records, and order lookup must still work.

Direction: the model does not need the key. What needs the key is the server code that runs the tool. Tools: build_system, run_tool, jload("config.json"). Concretely: remove the key line from the system text, and change run_tool to read the configuration instead of digging through the system text. The grader makes a new key and looks at the responses and every file other than config.json.

Keep the records, but masked

Fix log_request so that one line per request (rid and user name) is still left in data/app.log, but X-Token values, phone numbers, emails, and the partner key are not left verbatim. Keep only the last four digits of phone numbers (***-****-1234).

Direction: turn it off and you cannot investigate; write everything down and the record becomes a leak path. The job is choosing what to keep. Tools: re.sub, json.dumps. Concretely: do not keep the headers whole; record identity as a name, and mask the numbers and emails in the question, context, and answer with regular expressions. There is no reason to record the system text.

Detect prompt leakage with a canary

Fix build_system and check_output so that the canary value from config.json is put into the system text, and if that value shows up in a response, block the whole answer and leave a JSON line in data/audit.log containing that request's rid and "event": "canary". Normal answers must still go out.

Direction: a canary is a marker you put in place of a secret. If it shows up in output, the system text has leaked, so do not just delete the canary; block the whole answer. Tools: build_system, check_output, audit(). Concretely: read the canary from the configuration every time, and normalize whitespace and letter case before comparing. Also keep in mind the limit that a real model could slip past by rephrasing.

Filter other people's tickets before putting them in

Fix find_tickets so that the tickets put into the model are filtered to those of the requesting customer only. Even when asked with words from someone else's ticket, other people's information must not get mixed into the answer, and it must still answer with the customer's own tickets.

Direction: what went into the input is something the model can say. Apply permissions at the retrieval stage. Tools: the user of find_tickets, the owner in tickets.json. Concretely: remove tickets with a different owner before scoring.

Incident: three customers' numbers were found outside

Running incident ls-secret-leak raises the incident. The phone numbers of three customers were found outside the company: the found fragments are in /srv/ls-support/found.txt, the bot is /srv/ls-support/app.py, the data and records are in /srv/ls-support/data/, and the diagnostic bundle sent to a partner last week is /srv/ls-support/support-bundle.log. Find the path by which each customer's number went out and clean up the source: correct a wrongly recorded ticket owner to the owner of the number, take the number out of the system text, and mask only the numbers in the diagnostic bundle while keeping its lines. The code /srv/ls-support/app.py must block every defect you fixed in steps 3–6 of the lab (the key in the tool layer, masked records, the canary, per-customer retrieval; the grader sends those attacks again). In /root/incident/ls-secret-leak/report.txt, write three lines <customer>: <code> and evidence: <log line> (one line of the records). Each customer's code is one of the following: cross-tenant (another customer's ticket mixed into an answer), log-bundle (the diagnostic bundle sent to a partner), misfiled-ticket (a ticket with the wrong owner recorded), prompt-extract (extraction of the system text).

Direction: the symptom is the same for all three: "the number is outside". The path is which record the found fragment, exactly as shaped, could have come from. Tools: grep -F, jq, the owner in tickets.json, the phone in users.json. Concretely: grep for the fragment as is and see whether it is an answer in app.log or a question (q) in the diagnostic bundle. If it is an answer, compare the person who made that request and the owner of the ticket that went into the context with the real owner of the number (users.json); that separates another customer's ticket getting mixed in from a ticket with the wrong owner recorded.