When data becomes an instruction — direct and indirect prompt injection

In one sentence

A model cannot reliably tell which text in its input is a command and which is data. That is why "what is allowed to happen" must be decided not by the model but by a server policy outside the model.

Why this was needed

Think of SQL injection and it becomes easy. Code and data were mixed in one string, so data was executed as code, and we answered with placeholders (parameterization) that create a boundary the engine enforces between the two. LLMs have no such boundary. The system text, the user's question, the retrieved documents, and the tool results are in the end one sequence of tokens strung together, and a marker saying "data starts here" is only a tendency the model learned in training, not a rule that is enforced.

A user mixing instructions into their own question is direct injection. Trickier is indirect injection. If sentences that look like instructions hide inside documents the app retrieved, web pages or emails it was asked to summarize, or results a tool returned, the attacker does not even need to talk to our users; planting a single document is enough. The more RAG and tools an app has, the wider this road becomes. The OWASP Top 10 for LLM Applications 2025 lists this as the first risk (LLM01 Prompt Injection).

How it works

The labs in this course use a fake model that moves by rules instead of an external model. It must always give the same answer to the same input so that the grader can measure "what happened when the attack was sent" with the same result as many times as needed. This model is a caricature that exaggerates the weaknesses of a real model.

Property of the fake model	Its counterpart in a real model
Follows an instruction marker wherever it is in the input	Role separation is only a learned tendency, so it sometimes follows instructions inside data
Reads full-width, spaced, mixed-case, and base64 forms as the same instruction	Understands rephrased instructions, other languages, and encoded instructions too
Emits tool calls as lines of text	A function call is, in the end, output the app interprets as well

There are limits to state honestly. A real model is probabilistic. The same attack may work once in ten tries, or may work once rephrased. If "a filter that strips markers" worked against the fake model, that only means the filter knew all of this model's grammar; it is not evidence that the same filter blocks a real model. This is why passing a few tests does not prove safety.

So the center of the defense sits outside the model.

• Least privilege: do not execute the action the model produced as is. The server decides the recipient, the target, and the arguments with the privileges of the user who made the request.
• Human confirmation: hold high-risk actions coming from an answer mixed with untrusted text, and execute them once, only when the requesting user confirms.
• The server attaches sources: show not the citations the model wrote but the documents the server actually put into the input, with their trust levels.
• Filters and separator markers are a supporting layer: they reduce the volume of attacks and serve detection, but you do not trust them as a boundary.

Why these layers must be "outside" the model becomes clear when you picture failure. If a filter leaks, the model follows the instruction, and at that moment what remains last is the server's rule. Conversely, if the server's rule leaks, it is a defect in code, so tests can find it, and once fixed it does not leak again. Put a deterministic component in front of a probabilistic one: this is the principle that runs through this whole course.

The entrances for indirect injection are more numerous than you might think. Not only retrieved documents, but emails users paste in, memos saved from past conversations, the results of tools that read web pages, and summaries handed over by other agents all become model input. Think of it this way: every time you attach one more "memory" or "tool" to an app, one more door opens through which text written by others comes in.

What it looks like in the field

Typical incidents are an internal assistant that summarizes documents uploaded by customers sending email as a sentence in the document says, or a web page summarizing tool shipping conversation contents to an outside address as an instruction in the page says. In the logs the shape is always the same. Text from outside went into the prompt of some request, and in that very request an action nobody asked for happened. That is why, in operations, keeping a record of what went into the prompt and from where (request id, part, source) becomes the starting point of an investigation.

An investigation always climbs backward from the side effect: from one email that went out to the request id, from the request id to that request's input parts, and from the parts to the source of that text (document, memo, question, configuration). If the record has no parts and sources, this chain breaks and stops at "the model behaved strangely". That is why prompt records must be designed as audit material, not as debug dumps — with sensitive information masked, as the next module shows.

That does not mean you should throw away filters and separator markers. If you wrap outside text with a marker such as "what follows is data" and flag sentences that look like instructions, the probability that a real model follows them usually goes down. But going down and being blocked are different things. Use the supporting layer for detection and alerts; permission for actions always comes from the server's rules.

What you will do in the next lab

You open an internal document assistant, read from the records the input the model actually received, find in the pile of documents every hidden instruction in its different forms, and reproduce the attack once. Then you add, in turn, a recipient rule, holding and confirmation, and sources attached by the server, and at the end you untangle an incident in which the same symptom (mail sent outside the company) arose by different paths in three departments.