LLM App Security — From Injection to Output Handling

When model output flows into SQL, HTML, and the shell — and the incident that ties the course together

Goal

Fix a report assistant that passes the model's output as is to SQL, HTML, the shell, and database writes with defenses on the receiving side, and untangle a coupon leak incident that mixes the paths of the whole course.

Why it matters

The model's output is a value an attacker can write (the injection of the previous modules). Pass that value as is to the next system, and SQL injection, XSS, and command injection come back by way of the model. The defenses are applied not to the model but to the receiving side: having the model pick by name a query the server holds, placeholders, read-only connections, escaping that fits the context, execution without a shell, and schema validation. The fake model in this lab always outputs the attacker's text when there is an instruction, and a real model produces such output occasionally. Receiving-side defenses give the same result whenever that occasion comes. Every time it grades, the grader starts the app with a new database, coupons, and markers, and checks whether the database changes, coupons come out, tags are drawn, or marker files appear, and also whether the results of normal questions are right.

Steps

Unpack the materials with python3 /opt/fixtures/llmsec/kit.py report, then start /root/llmsec/report/app.py on 127.0.0.1:8104 (no 0.0.0.0, and keep it alive even if you close the terminal). The seed in curl 127.0.0.1:8104/health must be the same as the seed you received when unpacking.
Look at what follows sql= (the SQL the model produced) in the query record /root/llmsec/report/data/history.log, and write the number of entries that are not a single SELECT statement (several statements, or not a SELECT) and the number that touched the coupons table in /root/llmsec/report/flow.txt as two lines, non_select=<count> and coupon_reads=<count>.
Fix plan_query and /root/llmsec/report/templates.txt so that the model's output is not executed as SQL: the model picks by name one of the queries the server holds, values are passed as parameters, and the connection is opened read-only. The results of the three questions "total sales by region", "total sales by month", and "top five customers" must stay the same as now.
Fix render_page so that the question (q) of /page and the model's answer are escaped before being inserted into HTML. Tags must show as text (not deleted), and normal help answers must still show as they are.
Fix export_file so that the file name the model picked is validated by a rule (letters, digits, -, _, 1–40 characters, plus .csv) and copied to data/out/ without going through a shell. Names that fit the rule must still be exported.
Fix save_note so that the memo the model summarized is validated against a schema (a string, 1–300 characters, a customer that exists in customers) and then saved with placeholders. If it does not fit, do not save it and answer with a 4xx. A normal memo containing a single quote must also be saved exactly as written.
Running incident ls-coupon-leak raises the incident. Three coupon codes were used outside (/srv/ls-report/found.txt). Work out the path for each coupon owner, fix /srv/ls-report/app.py as in steps 3–6 of the lab, remove the sources, replace only the leaked coupons with new codes, and then write three lines <coupon-owner>: <code> and evidence: <log line> (one line of the records) in /root/incident/ls-coupon-leak/report.txt. The code is one of log-bundle (the record bundle sent to a partner), prompt-extract (extraction of the system text), shell-export (a shell command in an export name), or sql-from-model (SQL the model wrote).

Notes

Start an app with python3 /root/llmsec/<app>/app.py. Keep it in the background so it stays alive even if you close the terminal (setsid nohup … > <file> 2>&1 &). Every time you make a fix, stop the running one and start it again; the step 1 grader checks whether the running app was started from the current app.py (a check that imitates the operations accident where code from before the fix keeps receiving requests). It must still be running at the end for step 1 to be graded again.
Requests: curl -s -H "X-Token: <token>" -H 'Content-Type: application/json' -d '{"q":"…"}' 127.0.0.1:<port>/<path>. The tokens are in data/users.json.
The grader does not use the app you started; it copies the app.py file into a neighboring directory, starts it the same way you do with the new data/ inside (random data the grader made) and --port, and sends random attacks. Do not delete the lines that read data relative to the data/ next to the app and the port from --port. The functions to fix are listed in the file header. Code that notices it is being graded and behaves differently counts as a failure.
The fake model is /opt/fixtures/llmsec/fakellm.py. Read that file's header and directives() to learn what it reads as an "instruction".
Endpoints: POST /query {"q"}, GET /page?q=… (with the X-Token header), POST /export {"q"}, POST /note {"customer", "q": "note: …"}.
The sqlite3 CLI is not in this image. Look at the database with python3 -c 'import sqlite3; …'.
Incident tools: incident list shows the incidents you can raise and a one-line symptom for each, and incident show <name> shows the ticket again. Raising the same incident again puts it into its original broken state with a new seed (the cause for each target is chosen again too).
The report is /root/incident/<name>/report.txt. Write one line <name>: <cause-code> per target, and at the end add evidence: <log line> (one line of the records that showed the cause) as is (a whole line from a log or configuration, 20 characters or more). The grader first sends new random attacks to the fixed code again (as a regression, not only this incident's paths but also the attacks from the earlier steps of the same lab), checks that a different cleanup was done for each cause, and only then looks at the report. The evidence line must be a line that was in the records right after the incident, and it must connect to that target's incident request or the value that leaked.
Common mistake: filtering by looking for words like DROP and UPDATE in the SQL string. You have to chase variations forever; make sure the model does not write SQL, and let the engine (read-only) block it.

Unpack the app and start it on loopback

Unpack the materials with python3 /opt/fixtures/llmsec/kit.py report, then start /root/llmsec/report/app.py on 127.0.0.1:8104 (no 0.0.0.0, and keep it alive even if you close the terminal). The seed in curl 127.0.0.1:8104/health must be the same as the seed you received when unpacking.

Direction: a lab app has no reason to be open to the outside; the address it listens on is its exposure. Tools: ss -ltnp, curl, setsid and nohup. Concretely: the unpack command is exactly as above. After starting it, check that port 8104 shows as 127.0.0.1 in ss -ltn and that the seed in /health is the same as /root/llmsec/report/data/seed.txt.

Count the SQL the model produced

Look at what follows sql= (the SQL the model produced) in the query record /root/llmsec/report/data/history.log, and write the number of entries that are not a single SELECT statement (several statements, or not a SELECT) and the number that touched the coupons table in /root/llmsec/report/flow.txt as two lines, non_select=<count> and coupon_reads=<count>.

Direction: what the model made the system execute is already in the records. Count how often "the model wrote it, so it's fine" was wrong. Tools: grep, awk -F'sql=', python3. Concretely: split statements on semicolons, drop the empty ones, and count. The two numbers can overlap (there is also a single SELECT that reads coupons).

The model only picks a query

Fix plan_query and /root/llmsec/report/templates.txt so that the model's output is not executed as SQL: the model picks by name one of the queries the server holds, values are passed as parameters, and the connection is opened read-only. The results of the three questions "total sales by region", "total sales by month", and "top five customers" must stay the same as now.

Direction: as long as the model writes SQL, that SQL may be the attacker's. The server holds the SQL, and you change things so the model picks a name. Tools: templates.txt, a dictionary (name → SQL), sqlite3.connect(f"file:{path}?mode=ro", uri=True), the placeholder ?. Concretely: change what follows the | in templates.txt from SQL to a name, and make plan_query execute the server's SQL only when that name is in the dictionary. An unknown name is a 4xx.

Escape before inserting the answer

Fix render_page so that the question (q) of /page and the model's answer are escaped before being inserted into HTML. Tags must show as text (not deleted), and normal help answers must still show as they are.

Direction: the browser does not know where text came from. Convert both the model's answer and the echoed question to fit the context of the insertion point. Tool: html.escape. Concretely: wrap both places. Deleting tags breaks normal characters too and misses variations.

Names by rule, copying without a shell

Fix export_file so that the file name the model picked is validated by a rule (letters, digits, -, _, 1–40 characters, plus .csv) and copied to data/out/ without going through a shell. Names that fit the rule must still be exported.

Direction: the moment you hand a string to the shell, that string's grammar becomes the shell's grammar. Do jobs that need no shell without a shell. Tools: re.fullmatch, shutil.copyfile, an argument list for subprocess. Concretely: if a name does not fit the allow rule, refuse it with a 4xx (do not fix it up and accept it). If the rule has no path separator, path escape is blocked as well.

Save with placeholders only what passed the schema

Fix save_note so that the memo the model summarized is validated against a schema (a string, 1–300 characters, a customer that exists in customers) and then saved with placeholders. If it does not fit, do not save it and answer with a 4xx. A normal memo containing a single quote must also be saved exactly as written.

Direction: do not trust that the model kept to the format. Check type, length, and references, and pass values outside the SQL string. Tools: isinstance, len, SELECT 1 … WHERE name = ?, INSERT … VALUES (?, ?). Concretely: executescript does not accept placeholders; switch to execute. The customer name also comes from the request, so treat it the same way.

Incident: three coupons were used outside

Running incident ls-coupon-leak raises the incident. Three coupon codes of the report assistant (production version /srv/ls-report/app.py, data /srv/ls-report/data/) were used outside (found fragments /srv/ls-report/found.txt, records data/app.log, bundle sent to a partner /srv/ls-report/logs/vendor-bundle.log). Work out the path for each coupon owner, fix /srv/ls-report/app.py as in steps 3–6 of the lab (queries, screen, export, memos; the grader sends all those attacks again, and normal use must still work), and remove the sources: take the coupon out of the system text, mask only the coupons in the partner bundle while keeping its lines, and delete files that appeared in the public directory (data/public/). Then, in the coupons table, replace only the leaked coupons with new codes (CP- followed by 6 or more uppercase letters and digits, not overlapping other coupons), keeping the rows and leaving coupons that did not leak as they are. In /root/incident/ls-coupon-leak/report.txt, write three lines <coupon-owner>: <code> and evidence: <log line> (one line of the records). Each owner's code is one of the following: log-bundle (the record bundle sent to a partner), prompt-extract (extraction of the system text), shell-export (a shell command in an export name), sql-from-model (SQL the model wrote).

Direction: the exits you saw in the course have gathered in one place: input (injection), secrets (system text, records), and output (SQL, shell). The shape of a fragment tells you which exit it passed through. Tools: grep -F, jq, kind, sql, cmd, and answer in app.log, python3 -c 'import sqlite3 …'. Concretely: search for each fragment as is: if it has the shape of a query result row, look at the sql of kind=query; if it is a line of the system text, the answer of kind=page; if it is a CSV line, the cmd of kind=export and data/public; if it is a question sentence, the partner bundle. The fixes follow the same thinking as steps 3 and 5 of the lab, and replacing coupons is done one at a time with UPDATE … WHERE owner = ?.