← Back to digest
AI Security

"An attack does not tell the model what to do"

What's this about?

The single most transferable sentence in the writeup: "An attack does not tell the model what to do. The attack just makes the malicious path the one worth pursuing." The 415 response contains no instruction. It simply makes `curl` the obvious next thing to try, and Claude's own reasoning does the rest — "WebFetch got a 415. Let me try directly."

Why does this defeat injection filters?

Because there is nothing to filter. Prompt-injection defences look for imperative content in untrusted input: instructions, role-play, "ignore previous". A status code is not an instruction. Neither is a file layout, a plausible archive structure, or an error message. The attacker is shaping the environment so that the model's ordinary problem-solving arrives at the desired action on its own — and a defence that classifies text cannot see an incentive gradient.

Why does this matter to you?

  1. It reframes prompt injection as environment design, which means content filtering is structurally incapable of covering it.
  2. It is the practical version of the "when tool outputs become commands" paper from a few days ago: the missing control is authorization at the point of action, not detection at the point of input.
  3. If you are threat-modelling an agent, the question to ask is not "could someone inject text here" but "could someone make a bad path look like the reasonable one".
← The Auto Mode attack chain, step by step"A single layer of defence where we needed several" →