Prompts, logs, and answers are all exits — sensitive information disclosure

In one sentence

Whatever the model can see may someday come out in an answer, and every place that writes down the prompt copies its contents. Do not put secrets in, filter other people's data before putting it in, mask the records, and detect what still leaks with markers.

Why this was needed

There are three roads by which sensitive information leaks from an LLM app. First, the system text. During development it is common to write an API key or internal contacts into the system text "so the model can use them when calling a tool", but the system text is not a secret store. Requests that make the model repeat that text vary endlessly, and there is no guarantee the model always refuses them. The OWASP Top 10 for LLM Applications 2025 lists system prompt leakage (LLM07) as an item separate from sensitive information disclosure (LLM02).

Second, logs. If you write prompts, headers, and answers down verbatim so you can look at them again during an outage, everything that went into the prompt (customer phone numbers, authentication tokens, keys) is copied into the logs. Logs go to more people and systems than the service does: log collectors, diagnostic bundles, partners, and long-lived backups.

Third, retrieval permissions. If you find tickets similar to the question without regard to which customer they belong to and put them in, the model answers with another customer's personal information. Writing "do not tell anyone else's information" in the system text is not a boundary; it is trusting that the model will not say what has already gone into its input.

How it works

Layer	What it does	Does it trust the model
Don't put it in	Keys are read only in the server code where the tool runs	No
Filter	Only the requesting user's data is a retrieval candidate	No
Mask	Records show a name instead of a token, and only the last four digits of a phone number	Not applicable
Detect	Put a canary (a marker that must never leak) in the system text, and block and record it if it shows up in output	No

Canaries must be treated honestly. The fake model in this course appends the system text verbatim when told to "show the system text", so a canary string check catches it. A real model can rephrase, translate, or emit it one character at a time, and then the string check misses it. So a canary is a detection device, not a guarantee, and the first priority is always not to put secrets into the model's input.

Spelled out concretely, "don't put secrets in" means this. All the model needs to call a tool is the tool's name and arguments; the key the tool uses to call an outside API is read on the spot by server code from configuration or a secret store. It is enough to tell the model "there is a tool that looks up order status". Split this way, even if the whole prompt leaks, the key does not leak with it, and you do not need to edit the prompt when you rotate the key. For tools that need per-user permissions, the server likewise carries that user's identity without going through the model.

Masking has principles too. If you turn records off, you cannot investigate outages, so you do not turn them off; you choose what to keep. Identity as a name rather than a token, only as much as is needed to verify a customer (the last four digits of a phone number), and always the request id, so the record can be connected with other records. The OWASP Logging Cheat Sheet also recommends not recording session ids, access tokens, or personal information, or masking them.

Where you mask matters too. Mask in one place, the function that writes the records. If you decide to mask separately when showing on screen and when sending a bundle to a partner, someday one place will be missed. And records already left verbatim are not solved by "we will mask from now on"; you must find the current files and copies already sent out and recreate them. That is why the lab's incident demands this cleanup as well.

What it looks like in the field

Things like a customer support bot's diagnostic logs being sent to a partner with customer phone numbers and login tokens inside verbatim, "VIP customer contacts" written in the system text going out to the screen with a single prompt extraction request, or a consultation record attached to the wrong customer account so that one customer's number went out in an answer to another customer. On the surface they are all the same, "a number was found outside", and the investigation is matching the found fragment one by one against the records it could have come from. Each path has a different place to block and a different place to clean up, so if you pick the wrong path, you fix the wrong place and the real exit stays open.

One more point about retrieval permissions. Apply the customer separation "before" retrieval, not "after". If you score and pick the top few and then remove other people's tickets, other people's tickets take up the slots and push your own tickets out, and if the filtering code is off by a single line, other people's data goes straight into the input. It is safer to put "only the requesting customer's" into the query that picks the candidates itself; in a real system using vector search, that is the place for a metadata filter.

What you will do in the next lab

You count the secrets left verbatim in the customer support bot's records from yesterday, move the key from the system text to the tool layer, mask the records, block prompt leakage with a canary, and filter retrieval per customer. In the final incident, you trace back from the fragments how three customers' numbers leaked by different paths.