Making an AI better has meant making it bigger for about six years now, and it has worked well enough that most people treat it as a rule, but a recent paper from MIT found the point where it stops. Every LLM has to hold more concepts than it has room for, so it stacks them into the same directions and lets them overlap.
That overlap is called superposition, and it is the reason these models work as well as they do, though it is also the reason they get things wrong. The paper worked out the exchange rate.
Double the width of a model and you halve the errors, which turns out to be most of why bigger models have been better all along. In this video, I break down what a model's internal space actually looks like, why holding more concepts than it has dimensions is what makes models wrong in the first place, and the three separate things that stop this strategy from running forever, only one of which is money.
🧭 Chapters
- 00:00 The Space Inside GPT-1, 2 and 3
- 00:40 How LLMs Actually Learn
- 01:24 More Concepts Than Dimensions
- 01:54 Superposition Explained
- 02:33 Where Interference Comes From
- 03:08 The One Over Width Formula
- 04:16 Why Every Doubling Costs Six Times More
- 05:02 Zipf's Law Locks The Rate
- 06:00 Why The Errors Never Reach Zero
- 06:37 The Ceiling
- 07:18 What Researchers Are Building Instead
📚 Extra resources
- Superposition Yields Robust Neural Scaling (MIT, NeurIPS 2025): https://arxiv.org/abs/2505.10465
- Toy Models of Superposition (Anthropic, 2022): https://transformer-circuits.pub/2022...
- Language Models are Unsupervised Multitask Learners (GPT-2, 2019): https://cdn.openai.com/better-languag...
- Language Models are Few-Shot Learners (GPT-3, 2020): https://arxiv.org/abs/2005.14165
- Training Compute-Optimal Large Language Models (Chinchilla, 2022): https://arxiv.org/abs/2203.15556
- The rising costs of training frontier AI models (Epoch AI, 2024): https://arxiv.org/abs/2405.21015
