AirLLM: 70B inference on a 4GB GPU

What's this about?

AirLLM runs 70-billion-parameter models on a single 4GB consumer GPU by loading the network layer-by-layer from disk instead of holding it all in memory — slow, but it runs.

What exactly is the memory problem?

A 70B model needs ~140GB in standard precision — normally sharded across multiple $30k datacenter GPUs. Since a transformer executes layers one at a time, you can stream each layer in, compute, and evict it. The trade: latency balloons because disk I/O sits on the critical path.

What's the community saying?

1. Main dispute: tokens-per-second — at seconds-per-token it's a demo, not a daily driver.
2. Fans see offline/air-gapped value: big-model access with no cloud and no budget.
3. Comparisons to llama.cpp partial offloading and mmap tricks — is this new or a repackaging?