What's this about?
Philip Kiely and Ali Taha of Baseten join Latent Space for a deep, practical walk through inference engineering — the craft of serving large models fast and cheaply in production.
What is "inference"?
Training is teaching a model (expensive, one-time); inference is running the finished model to answer requests (happens constantly, and is where most real-world cost lives). "Inference engineering" is the unglamorous art of making that serving cheap and low-latency — batching, caching, quantization, GPU scheduling.
Why does it matter?
As AI products scale, inference cost, not training cost, is what makes or breaks the business. This is the operational knowledge behind why some AI apps are profitable and others bleed money.