What's this about?
Computer-use agents are usually improved by making them better at reading screenshots and choosing where to click. StateAct argues a screenshot is a lossy picture of the real thing, and that feeding the agent the underlying program state directly yields better results on long-horizon tasks (18 upvotes).
What exactly is "program state"?
Program state is the actual structured data behind what you see — the list of open files, field values, the DOM of a page — as opposed to the rendered image of it. Reading state instead of a screenshot is like getting the spreadsheet's numbers directly rather than photographing the screen and squinting at the pixels.
Why does it matter?
Screenshot-based computer use is brittle over many steps; small misreads compound. Grounding agents in program state where available is a cheaper, more reliable path for the long-horizon tasks (multi-step form-filling, tool use) that agents most often botch.