LLM App Security — From Injection to Output Handling

Whose privileges run the tool the model picked — roles, allowlists, paths, and confirmation

Goal

Bind a file assistant that executes the tool the model picked as is with service privileges to the requesting user's role, a tool allowlist, path validation, and a person's confirmation.

Why it matters

An agent's tools run with the app server's privileges, and the model does not tell the user's words apart from words in documents. So a viewer's request or a single sentence in a shared document can execute a deletion (the confused deputy). The fake model in this lab makes that weakness always visible; a real model does it only "occasionally", and since tests cannot find every occasion, permissions are applied outside the model. Every time it grades, the grader starts the app with new users, files, and instructions and checks whether files are actually deleted, written, or read. Normal requests (a viewer reading, an editor writing, an admin's confirmed deletion) must still work.

Steps

Unpack the materials with python3 /opt/fixtures/llmsec/kit.py agent, then start /root/llmsec/agent/app.py on 127.0.0.1:8103 (no 0.0.0.0, and keep it alive even if you close the terminal). The seed in curl 127.0.0.1:8103/health must be the same as the seed you received when unpacking.
In the past execution record /root/llmsec/agent/data/history.log, look only at successful (result=ok) executions, and write the number of executions of tools not in the role's policy table and the number that touched paths outside the user's home (going above the home with .., or absolute paths) in /root/llmsec/agent/review.txt as two lines, violations=<count> and escapes=<count>. The policy table is in the header of app.py.
Fix allowed so that tool execution is permitted by the requesting user's role (exactly as in the policy table; a tool not in the table is not allowed for anyone). Record refusals in data/audit.log as "event": "denied". A viewer's reading and an editor's writing must still work.
Fix offered_tools so that the model is shown only the tools that user's role may use. A viewer's delete request must not produce a delete call at all, and no one's request may produce run_shell. Even if the model makes up a name not on the list, it must not be executed and the app must not crash.
Fix resolve so that path arguments are resolved to real paths and used only when they are inside that user's home (data/files/<name>/). Other people's files must not be readable or writable via .., absolute paths, or symbolic links pointing outside the home, while paths inside the home (including ones like sub/../sub/x.txt) still work.
Fix run_call and handle_confirm so that delete_file is always held regardless of the confirm argument the model put in: put {"status": "pending", "id": <confirm-id>, ...} in the response actions, and delete only once when the requesting user sends POST /confirm {"id": ...}. Reject another user's confirmation and a second confirmation with a non-2xx code, and another user's confirmation attempt must not remove the hold.
Running incident ls-agent-overreach raises the incident. Three people's files have disappeared (/srv/ls-agent/missing.txt). Work out the reason for each person, fix the app /srv/ls-agent/app.py as in steps 3–5 of the lab, restore only the wrongly deleted files from the backup /srv/ls-agent/backup/files/, and then write three lines <owner>: <code> and evidence: <log line> (one line of the records) in /root/incident/ls-agent-overreach/report.txt. The code is one of model-confirm (an instruction in the context filled in the confirmation), owner-request (the owner confirmed and deleted it directly; not an incident), path-escape (a path that went into someone else's home), role-ignored (a deletion not allowed for the role), or shell-tool (the shell tool).

Notes

Start an app with python3 /root/llmsec/<app>/app.py. Keep it in the background so it stays alive even if you close the terminal (setsid nohup … > <file> 2>&1 &). Every time you make a fix, stop the running one and start it again; the step 1 grader checks whether the running app was started from the current app.py (a check that imitates the operations accident where code from before the fix keeps receiving requests). It must still be running at the end for step 1 to be graded again.
Requests: curl -s -H "X-Token: <token>" -H 'Content-Type: application/json' -d '{"q":"…"}' 127.0.0.1:<port>/<path>. The tokens are in data/users.json.
The grader does not use the app you started; it copies the app.py file into a neighboring directory, starts it the same way you do with the new data/ inside (random data the grader made) and --port, and sends random attacks. Do not delete the lines that read data relative to the data/ next to the app and the port from --port. The functions to fix are listed in the file header. Code that notices it is being graded and behaves differently counts as a failure.
The fake model is /opt/fixtures/llmsec/fakellm.py. Read that file's header and directives() to learn what it reads as an "instruction".
The body of /run is {"request": "...", "context": ["<shared document line>", ...]} (the context holds lines of documents shown along with the request). The context goes into the model's input too.
Incident tools: incident list shows the incidents you can raise and a one-line symptom for each, and incident show <name> shows the ticket again. Raising the same incident again puts it into its original broken state with a new seed (the cause for each target is chosen again too).
The report is /root/incident/<name>/report.txt. Write one line <name>: <cause-code> per target, and at the end add evidence: <log line> (one line of the records that showed the cause) as is (a whole line from a log or configuration, 20 characters or more). The grader first sends new random attacks to the fixed code again (as a regression, not only this incident's paths but also the attacks from the earlier steps of the same lab), checks that a different cleanup was done for each cause, and only then looks at the report. The evidence line must be a line that was in the records right after the incident, and it must connect to that target's incident request or the value that leaked.
Common mistake: blocking by looking for only .. in the path string. Absolute paths and symbolic links get through; resolve to real paths and compare.

Unpack the app and start it on loopback

Unpack the materials with python3 /opt/fixtures/llmsec/kit.py agent, then start /root/llmsec/agent/app.py on 127.0.0.1:8103 (no 0.0.0.0, and keep it alive even if you close the terminal). The seed in curl 127.0.0.1:8103/health must be the same as the seed you received when unpacking.

Direction: a lab app has no reason to be open to the outside; the address it listens on is its exposure. Tools: ss -ltnp, curl, setsid and nohup. Concretely: the unpack command is exactly as above. After starting it, check that port 8103 shows as 127.0.0.1 in ss -ltn and that the seed in /health is the same as /root/llmsec/agent/data/seed.txt.

Count out-of-role and out-of-home executions in the past runs

In the past execution record /root/llmsec/agent/data/history.log, look only at successful (result=ok) executions, and write the number of executions of tools not in the role's policy table and the number that touched paths outside the user's home (going above the home with .., or absolute paths) in /root/llmsec/agent/review.txt as two lines, violations=<count> and escapes=<count>. The policy table is in the header of app.py.

Direction: before fixing, count what has already happened. Judge the policy not by "what the model picked" but by "whose request it was". Tools: awk, grep, os.path.normpath in python3. Concretely: judge paths after normalizing them; a path that comes back inside the home is not outside even if it has .. in the middle. Do not count failed (result=error) lines.

Permit by the requesting user's role

Fix allowed so that tool execution is permitted by the requesting user's role (exactly as in the policy table; a tool not in the table is not allowed for anyone). Record refusals in data/audit.log as "event": "denied". A viewer's reading and an editor's writing must still work.

Direction: the tool name the model picked is only a suggestion. Whether to execute it is decided by the requester's role. Tools: allowed(user, tool), the role in users.json. Concretely: move the policy table into code, and use a closed default that refuses roles and tools not in the table.

Trim the tools shown to the model

Fix offered_tools so that the model is shown only the tools that user's role may use. A viewer's delete request must not produce a delete call at all, and no one's request may produce run_shell. Even if the model makes up a name not on the list, it must not be executed and the app must not crash.

Direction: a tool that cannot be picked will never be executed either. The check right before execution (step 3) and trimming the list you show are different layers. Tools: offered_tools, TOOLS, allowed. Concretely: filter the list to show by role. Also check that the execution-side check refuses made-up names.

Resolve paths to real paths, and only inside the home

Fix resolve so that path arguments are resolved to real paths and used only when they are inside that user's home (data/files/<name>/). Other people's files must not be readable or writable via .., absolute paths, or symbolic links pointing outside the home, while paths inside the home (including ones like sub/../sub/x.txt) still work.

Direction: block with strings and a path written differently gets through. Look at where the filesystem actually points. Tools: os.path.realpath, os.path.commonpath, os.path.isabs. Concretely: resolve both the home and the target to real paths and check that their common ancestor is the home. If you raise the refusal as an exception, run_call returns it as an error.

Confirmation comes from a person, not the model

Fix run_call and handle_confirm so that delete_file is always held regardless of the confirm argument the model put in: put {"status": "pending", "id": <confirm-id>, ...} in the response actions, and delete only once when the requesting user sends POST /confirm {"id": ...}. Reject another user's confirmation and a second confirmation with a non-2xx code, and another user's confirmation attempt must not remove the hold.

Direction: confirm: true is an argument the model fills in, so an instruction in a document can put it in. Receive confirmation through a channel the model cannot write to. Tools: PENDING, secrets.token_urlsafe, handle_confirm. Concretely: when a delete call comes, instead of executing it, create an id and store it together with whose it is; on a confirmation request, take it out and execute it only if it is the requester's own, then delete it. Even after confirmation, it goes through the role and path checks again.

Incident: three people's files disappeared

Running incident ls-agent-overreach raises the incident. The files of three people who use the file assistant (production version /srv/ls-agent/app.py, data /srv/ls-agent/data/) have disappeared (list /srv/ls-agent/missing.txt, records data/audit.log and data/runs.log; runs.log keeps the request sentence, the confirmation a person sent (part=confirm), the context given along with it, and the call the model picked). Work out the reason for each person and fix /srv/ls-agent/app.py: block every defect from steps 3–5 of the lab (role, tool list, path), whether or not it relates to the incident's causes, and do not look at a confirm argument filled in by the model (leave as is the normal use where an admin adds "confirm": true to the /run request body to delete their own file). Then restore only the wrongly deleted files from the backup /srv/ls-agent/backup/files/ with the same content; do not restore what the owner confirmed and deleted directly. In /root/incident/ls-agent-overreach/report.txt, write three lines <owner>: <code> and evidence: <log line> (one line of the records). Each person's code is one of the following: model-confirm (an instruction in the context filled in the confirmation), owner-request (the owner confirmed and deleted it directly; not an incident), path-escape (a path that went into someone else's home), role-ignored (a deletion not allowed for the role), shell-tool (the shell tool).

Direction: even if "files disappeared" is the same, who it was, with what sentence and context, and which tool they produced differ. And not every deletion is an incident. Tools: grep, jq, delete and shell in audit.log, part=request, confirm, context, and call in runs.log, the role in users.json. Concretely: for each missing path, find in audit.log the record that touched that file and its rid, and look in runs.log at that rid's request sentence, context, and call. The forks in the road are whether the requester is the owner, whether that role may delete, and whether the confirmation came from the request field a person sent (the part=confirm line in runs.log) or from a tool argument the model filled in after looking at the context. Even if a context line contains a word like "confirm", it is not a person's confirmation.