LLM App Security — From Injection to Output Handling

Hidden instructions in RAG documents — actions are decided outside the model

Goal

Reproduce how a hidden instruction in an outside document retrieved by an internal document assistant sends mail, and fix it so that the recipient, holding and confirmation, and sources are decided by the server rather than the model.

Why it matters

A RAG app puts text written by others into the model's input. The model cannot reliably tell whether that text is data or instruction, so a single sentence that looks like an instruction can move the app's tools. The fake model in this lab exaggerates that weakness so it is always visible; a real model is probabilistic, so the same attack works only sometimes, which is why "I tested it a few times and it was blocked" cannot establish safety. The lab does not stop at filtering markers; it makes the server's rules decide the action whatever the model emits. Every time it grades, the grader sends freshly made attacks to the fixed app and also checks that normal requests still work.

Steps

Unpack the materials with python3 /opt/fixtures/llmsec/kit.py rag, then start /root/llmsec/rag/app.py on 127.0.0.1:8101 (no 0.0.0.0, and keep it alive even if you close the terminal). The seed in curl 127.0.0.1:8101/health must be the same as the seed you received when unpacking.
Send question 3 of /root/llmsec/rag/questions.txt as is to /ask, and write the ids of the documents that went into the model's input (data/prompts.log), in the order they went in, to /root/llmsec/rag/trace.txt as one line q3_docs=<id,id,...>.
Find every document in data/docs/ that hides a marker the fake model reads as an instruction, and write their ids in /root/llmsec/rag/poisoned.txt, one per line. The markers come in several forms.
Reproduce the attack once by sending a question that makes a document with a hidden mail instruction get retrieved, and write two lines in /root/llmsec/rag/repro.txt: mail=<spool-file> (the file name in data/spool) and to=<recipient> (that mail's recipient).
Fix run_call in app.py so that the mail recipient is allowed only as the requesting user's own address (users.json). For any other address, do not execute it, and leave a JSON line in data/audit.log containing that request's rid and "event": "blocked". A normal request to send to my own address must still go out.
Fix gate and handle_confirm so that if even one retrieved document is not trust: internal (including ones with no marker or another value), the action the model produced is not executed but held in the response actions as {"status": "pending", "id": <confirm-id>, ...}. Execute it only once when the same user sends POST /confirm {"id": ...}; reject another user's confirmation and a second confirmation with a non-2xx code; and another user's confirmation attempt must not remove the hold (so the owner can still confirm afterward). Actions from answers that used only internal documents are executed right away.
Fix sources_of so that the response sources is made not from the citation markers in the model's answer but from the documents the server actually put into the prompt: [{"id": ..., "trust": "internal|external"}].
Running incident ls-rag-exfil raises the incident. Last night, mail went out to addresses outside the company from the assistants of three departments (faq, hr, sales). Fix the code /srv/ls-rag/bot.py (shared by the three departments) so that no mail goes out to an address outside the company (not @corp.example), whatever path it comes by, clean up the source of the cause for each department, and then write faq: <code>, hr: <code>, sales: <code> and evidence: <log line> (one line of the records) in /root/incident/ls-rag-exfil/report.txt. The code is one of config-cc (a cc address in the configuration), memory-note (an instruction in a user memo), poisoned-doc (an instruction in a retrieved document), or user-prompt (an instruction in the user's question).

Notes

Start an app with python3 /root/llmsec/<app>/app.py. Keep it in the background so it stays alive even if you close the terminal (setsid nohup … > <file> 2>&1 &). Every time you make a fix, stop the running one and start it again; the step 1 grader checks whether the running app was started from the current app.py (a check that imitates the operations accident where code from before the fix keeps receiving requests). It must still be running at the end for step 1 to be graded again.
Requests: curl -s -H "X-Token: <token>" -H 'Content-Type: application/json' -d '{"q":"…"}' 127.0.0.1:<port>/<path>. The tokens are in data/users.json.
The grader does not use the app you started; it copies the app.py file into a neighboring directory, starts it the same way you do with the new data/ inside (random data the grader made) and --port, and sends random attacks. Do not delete the lines that read data relative to the data/ next to the app and the port from --port. The functions to fix are listed in the file header. Code that notices it is being graded and behaves differently counts as a failure.
The fake model is /opt/fixtures/llmsec/fakellm.py. Read that file's header and directives() to learn what it reads as an "instruction".
To send a question without shell quoting, it is convenient to build the JSON and pass it along, as in python3 -c 'import json,sys;print(json.dumps({"q":sys.argv[1]}))' "<question>" | curl … -d @- ….
Incident tools: incident list shows the incidents you can raise and a one-line symptom for each, and incident show <name> shows the ticket again. Raising the same incident again puts it into its original broken state with a new seed (the cause for each target is chosen again too).
The report is /root/incident/<name>/report.txt. Write one line <name>: <cause-code> per target, and at the end add evidence: <log line> (one line of the records that showed the cause) as is (a whole line from a log or configuration, 20 characters or more). The grader first sends new random attacks to the fixed code again (as a regression, not only this incident's paths but also the attacks from the earlier steps of the same lab), checks that a different cleanup was done for each cause, and only then looks at the report. The evidence line must be a line that was in the records right after the incident, and it must connect to that target's incident request or the value that leaked.
Common mistake: blocking it by just deleting the markers from the documents. This model is blocked that way, but in front of a real model, rephrased instructions remain. The grader even checks whether the blocked actions are left in the records.

Unpack the app and start it on loopback

Unpack the materials with python3 /opt/fixtures/llmsec/kit.py rag, then start /root/llmsec/rag/app.py on 127.0.0.1:8101 (no 0.0.0.0, and keep it alive even if you close the terminal). The seed in curl 127.0.0.1:8101/health must be the same as the seed you received when unpacking.

Direction: a lab app has no reason to be open to the outside; the address it listens on is its exposure. Tools: ss -ltnp, curl, setsid and nohup. Concretely: the unpack command is exactly as above. After starting it, check that port 8101 shows as 127.0.0.1 in ss -ltn and that the seed in /health is the same as /root/llmsec/rag/data/seed.txt.

Read the input the model actually received

Send question 3 of /root/llmsec/rag/questions.txt as is to /ask, and write the ids of the documents that went into the model's input (data/prompts.log), in the order they went in, to /root/llmsec/rag/trace.txt as one line q3_docs=<id,id,...>.

Direction: in RAG, most of the model's input is not text the user wrote but retrieved text written by others. Do not guess what went in; read it from the records. Tools: curl, grep, awk. Concretely: take the rid from the response, look at that rid's part=context lines in prompts.log, and collect the src= attached to each line in the order they first appear.

Find every hidden instruction in its different forms

Find every document in data/docs/ that hides a marker the fake model reads as an instruction, and write their ids in /root/llmsec/rag/poisoned.txt, one per line. The markers come in several forms.

Direction: the model reads more leniently than a filter. If you grep for one shape, you miss the ones with a different shape, and decoys that look similar are not instructions. Tools: directives() in fakellm.py, unicodedata, base64, grep. Concretely: first read that function to see how the fake model normalizes input and what it decodes and reads, then judge each document by the same rules.

Reproduce the attack once

Reproduce the attack once by sending a question that makes a document with a hidden mail instruction get retrieved, and write two lines in /root/llmsec/rag/repro.txt: mail=<spool-file> (the file name in data/spool) and to=<recipient> (that mail's recipient).

Direction: the attacker does not talk to our users. All it takes is for a user's ordinary question to pull in that document. Tools: the retrieval rules (retrieve), curl, ls data/spool. Concretely: pick one document containing a mail instruction and build a question from words in that document's body, and that document rises to the top. Match the response actions with the to of the spool file.

The server decides the recipient

Fix run_call in app.py so that the mail recipient is allowed only as the requesting user's own address (users.json). For any other address, do not execute it, and leave a JSON line in data/audit.log containing that request's rid and "event": "blocked". A normal request to send to my own address must still go out.

Direction: the argument the model produced (to) may be text an attacker wrote. What to do with that value is decided by the identity of the person who made the request. Tools: the user argument of run_call, audit(). Concretely: if the recipient differs from the user's email, do not send it, and record that you blocked it. The grader sends hidden instructions (in random forms), requests in which the user names an outside address directly, and normal requests, all together.

When untrusted text is mixed in, a person confirms

Fix gate and handle_confirm so that if even one retrieved document is not trust: internal (including ones with no marker or another value), the action the model produced is not executed but held in the response actions as {"status": "pending", "id": <confirm-id>, ...}. Execute it only once when the same user sends POST /confirm {"id": ...}; reject another user's confirmation and a second confirmation with a non-2xx code; and another user's confirmation attempt must not remove the hold (so the owner can still confirm afterward). Actions from answers that used only internal documents are executed right away.

Direction: even with the recipient tied to the user, an instruction in an outside document can make the app send the user an odd mail (such as a fake payment link). Bring a person in only at the risky moments. Tools: a PENDING dictionary, secrets.token_urlsafe, and going through run_call even after confirmation. Concretely: keep the confirmation id together with who created it, and delete it once used. If you reject someone else's id and a nonexistent id with the same response, not even whether an id exists leaks.

The server attaches sources

Fix sources_of so that the response sources is made not from the citation markers in the model's answer but from the documents the server actually put into the prompt: [{"id": ..., "trust": "internal|external"}].

Direction: "according to [kb-07]" is also the model's output. If an outside document impersonates an internal policy, people believe that citation. Tool: the docs that sources_of receives. Concretely: do not parse the answer text; copy the id and trust straight from the retrieval results. It is safer to treat a document with no marker as an outside one.

Incident: mail went outside the company from three departments

Running incident ls-rag-exfil raises the incident. Last night, mail went out to addresses outside the company from the assistants of three departments (faq, hr, sales) (spool /srv/ls-rag/bots/<dept>/spool/, records prompts.log and audit.log). Fix the code /srv/ls-rag/bot.py (shared by the three departments) so that no mail goes out to an address outside the company (not @corp.example), whatever path it comes by (both recipients and cc; normal mail to internal addresses goes out as before), and clean up the source of the cause for each department: remove the poisoned document from the retrieval targets, remove the poisoned memo from the memos, make the token of the account that injected the instruction stop working, and delete the outside cc from the configuration (leave documents and memos unrelated to the cause). In /root/incident/ls-rag-exfil/report.txt, write faq: <code>, hr: <code>, sales: <code> and evidence: <log line> (one line of the records). Each department's code is one of the following: config-cc (a cc address in the configuration), memory-note (an instruction in a user memo), poisoned-doc (an instruction in a retrieved document), user-prompt (an instruction in the user's question).

Direction: the symptom is the same in all three, but the place the instruction came in differs. Climb backward from one mail and look at that request's input part by part. Tools: grep, jq, the rid and cc in the spool, part= and src= in prompts.log. Concretely: open prompts.log with the rid of a spool file that contains an outside address and see which part (memory, context, user) held the instruction marker; if it went out with no marker, look at the config to see where the mail's cc came from. Each department also has decoys (instruction documents that were not retrieved, harmless instructions); only what connects to a mail that actually went out is a cause.