What's this about?
Caltech professor Anima Anandkumar set out to build the first open-source AI weather model and was met with scepticism: weather is chaotic, physics simulations have had decades of work and supercomputers, and the data isn't there. Within a year her team shipped FourCastNet, competitive with the best physics-based simulations, and short-range weather prediction became something you can do on a consumer GPU. The episode is about why physical systems resist the scaling playbook, and what replaces it.
What is a Neural Operator?
Standard neural networks learn a mapping between fixed-size inputs and outputs — a 256×256 image in, a label out. Physical systems don't have a fixed size: temperature is a continuous field, and the grid you sample it on is an arbitrary choice. A Neural Operator learns a mapping between *functions* rather than grids, so the same trained model works at different resolutions. That is what makes multi-scale physics tractable — you are modelling the underlying field, not one particular discretisation of it.
What's the community saying?
- The scaling argument breaks twice here. Open datasets in these domains have tens or hundreds of thousands of examples, not billions.
- And the resolution physics demands pushes context length into the hundreds of billions — in her words, "forget ever having a transformer for anything of this scale, all of the world's compute will not be enough."
- The alternative is unfashionable and effective: build in structure and inductive biases from physical law, rather than throwing tokens at it.
- Latent Space's own framing is that this is a large area the AI community has neglected and expects to grow fast.