What's this about?
Macaron-V1 is a concrete open approach to continual learning — models that keep improving after deployment — using self-improvement plus a "mixture-of-LoRA" design to add new skills without forgetting old ones. It's the practical companion to Dwarkesh's continual-learning thesis today.
What exactly is "LoRA"?
LoRA (Low-Rank Adaptation) is a cheap way to teach a big model a new skill: instead of retraining all its billions of weights, you train tiny add-on adapter layers and slot them in. "Mixture-of-LoRA" keeps several such adapters and combines them — so the model can accumulate new abilities as separate, composable pieces rather than overwriting itself.
Why is this notable?
It attacks the core obstacle to continual learning — catastrophic forgetting — with a modular, efficient trick. If adapters can be added indefinitely without degrading the base, that's a plausible path to models that genuinely keep learning.