← Back to digest
AI Security

When tool outputs become commands

What's this about?

Tool-augmented agents must consume untrusted runtime observations to do open-ended work. The paper's framing: when tool outputs stop merely providing data and start specifying concrete actions, they have effectively become commands — and the system has no mechanism distinguishing "this text induced an action" from "this action was authorized."

What is the distinction being drawn?

Action induction is what makes the agent want to do something. Runtime authorization is what decides whether it may. In a normal system these are separate: a user clicking a button induces, and an access-control check authorizes. In an agent, the text that induces the action and the reasoning that authorizes it are the same substrate — so a web page saying "now delete the staging bucket" is competing on equal footing with your instructions. Separating the two is the architectural fix.

Why does this matter to you?

  1. This is the cleanest formulation of why prompt injection is not a filtering problem: filtering tries to catch bad induction, when the missing piece is authorization.
  2. It suggests an implementable control — actions with real consequences pass through a check that never reads tool output.
  3. Directly applicable to any scanner or automation that ingests third-party content and then acts on what it read.
← Layered LLM defenses only compound if they fail independentlyCAITLYN: can agents synthesise their own injection defenses? →