What's this about?
Vision-language models score well on document benchmarks like DocVQA — but real documents mix length, layout complexity, modality, and question difficulty in ways those benchmarks don't isolate. SynthDocBench generates documents where each factor is controlled independently, exposing exactly where models break. HF's most-upvoted paper today (56↑).
What exactly is a "controlled benchmark"?
Ordinary benchmarks are grab-bags of real examples — when a model fails, you can't tell *why*. A controlled benchmark is like an eye exam: vary one variable at a time (longer document, denser layout, chart vs table) and measure where performance degrades. Synthetic generation makes this possible at scale.
Why is it worth noting?
Long-document understanding is the workhorse of enterprise AI (contracts, filings, medical records); a diagnostic that says *which* factor kills a model is directly actionable for anyone choosing one.