What's this about?
A paper arguing that AI agents controlling a computer shouldn't rely mainly on reading screenshots. Instead, feed them the underlying program state directly, which gives better results on long, multi-step tasks.
What exactly is a "computer-use agent"?
It's an AI that operates software the way a person would — clicking buttons, filling forms, navigating apps — to complete tasks like "book this flight" or "update this spreadsheet." Most current ones "see" the screen via screenshots and guess what's there. StateAct's point: a screenshot is a lossy picture of what the app actually contains, so handing the agent the real underlying data (the "program state") is more reliable than making it re-interpret pixels every step.
Why is this worth noting?
Screenshot-based perception is the dominant approach and its errors compound over long tasks. Bypassing pixels where possible is a pragmatic direction for making computer-use agents dependable enough for real work.