What's this about?
Agents enter executable sandboxes with unfamiliar rules, flawed manuals and feedback tools. They must experiment, infer the true system and solve held-out tasks.
What exactly is a verifiable alien world?
It is a made-up environment whose rules can be checked exactly by code but deliberately conflict with ordinary knowledge, reducing the chance that a model merely remembers an answer from training.
Why does this matter?
author-reported preprint; 12 HF upvotes.
- AlienCode and AlienLogic contain 140 tasks around 55 discovery targets.
- Strong systems sometimes learn and transfer unfamiliar rules successfully.
- More exploration can stall or undo earlier gains, showing that gathering evidence is not automatically cumulative.