← Back to digest
arxiv: Large Language Models

Temperature zero does not remove semantic-query run variance

What's this about?

Across 1.2 million calls, authors find row decisions changing across repeated executions at temperature zero, sometimes amplified into different final answers. Voting reduces but does not eliminate variation. Single-run benchmark comparisons can therefore overstate confidence.

What exactly is semantic query?

A database operation that asks a model to interpret meaning in records, rather than applying only exact comparisons.

What's the community saying / why does it matter?

No substantive discussion collected. Repeated measurements and variance reporting matter even when sampling is configured conservatively.

← RATED chooses tools using live reliability feedbackDeepMind and Biohub explain what AlphaFold still does not solve →