Model output is user input — when it flows into SQL, HTML, and the shell

In one sentence

When the model's output flows into SQL, a browser, a shell, or a database write, old injection attacks come back by way of the model. Treat model output as just as untrusted as user input, and apply the defense that fits the receiving side's context (choosing an intent, parameters, escaping, execution without a shell, schema validation).

Why this was needed

As the previous three modules showed, an attacker can write the model's output: through instructions in documents, the user's question, or tool results. Yet many apps treat the model's output as "a value our system produced" and pass it on to the next stage as is. A report assistant that turns natural language into SQL and runs it, a screen that inserts the model's answer into HTML, an export that builds a shell command with a file name the model chose, and an INSERT that strings together a memo the model summarized are examples. Then the old problems of SQL injection, XSS, and command injection reopen under the illusion that "the model wrote it, so it's fine". The OWASP Top 10 for LLM Applications 2025 lists this as Improper Output Handling (LLM05).

How it works

Where it flows	Don't	Do
SQL reads	Execute SQL the model wrote	Have the model pick one of the queries the server holds "by name", pass values as parameters, and open the connection read-only
HTML	Insert the answer as is	Escape to fit the context of the insertion point (body, attribute); the same applies where the question is echoed back
Shell	Concatenate strings and run with shell=True	Use functions that do not go through a shell (file copy) or argument lists, and restrict names with an allow rule (letters, digits, -, _, and the extension)
SQL writes	Concatenate values inside quotes	Placeholders, and before that, schema validation (type, length, whether referenced values exist)

The read-only connection deserves special emphasis. Checking SQL strings to filter out DROP and UPDATE means chasing variations forever, but on a connection opened with mode=ro, the engine refuses writes. In being a boundary the engine enforces rather than a string check, it is the same kind of defense as parameterization.

Schema validation means not trusting "whether the model kept to the format". A string can come where a number should be, 1,000 characters where 300 should be, a nonexistent name where an existing customer should be. If it does not fit, do not execute it and refuse with a 4xx, rather than crashing with a 500 or fixing it up and accepting it.

One more thing: escaping depends on "where you put it". The HTML body, attribute values, JavaScript strings, and URLs each have different dangerous characters, which is why OWASP's XSS prevention guidance says to choose the encoding that fits the context of the insertion point. This lab covers only the body context, but if a design puts model answers into link addresses or scripts, the same function is not enough. It is better not to put model output in such places at all.

The shell is the same story. A quoting function (shlex.quote) blocks command injection but does not block names that climb directories with ... So narrow names with an allow rule, and do jobs that need no shell (copying files) without a shell. The reason "reject if it does not match the rule" is sturdier than "fix up dangerous characters and accept" is that the fixing side must know every case of input the attacker chooses, while the rejecting side only needs to know the shapes it allows.

The fake model in this course always outputs the attacker's text as is when there is an instruction. With a real model, such output comes occasionally and unpredictably. Output handling defenses are applied on the receiving side so that the result is the same whenever that "occasionally" comes, which is why they can be tested regardless of how well behaved the model is.

What it looks like in the field

Typical cases are a natural language query tool that returned an out-of-permission table, that is, another customer's data, through a query the model built; a chat screen that drew the tags in a model answer as is, so a request went out from the user's browser to an outside address; and a "save the report under this name" feature that executed shell syntax in the name. The final incident of this lab ties the whole course together. Three coupon codes were used outside, and four candidate paths appear: SQL the model wrote, extraction of the system text, an export shell command, and a record bundle sent to a partner. Three of them are the real paths.

If you add receiving-side defenses, change the records too. Records that used to write down the SQL, HTML, and commands the model produced verbatim become a store of attack strings in themselves, and if another tool reads those records again and executes them, the same problem happens once more. After fixing, keep only what is needed for judgment, such as the intent name the model picked, the reason for a refusal, and the request id.

What you will do in the next lab

You count the dangerous SQL the model produced in the report assistant's query records, and fix queries with intent selection, parameters, and read-only connections, the screen with escaping, exports with a shell-free copy and a name rule, and memo saving with schema validation and placeholders. Finally, you work out and block the three paths of the coupon leak incident and replace only the coupons that leaked.