Kaplan Scaling Laws, January 2020: ‘Just Add Compute’

---

The Finding

The Kaplan scaling laws paper showed that the loss of a language model — the measure of how poorly the model predicts the next word, used as the standard performance metric in language modelling — decreases as a power law (a mathematical relationship in which one quantity scales as a power of another) as you increase the model’s parameters, the training data, or the computation. A power law means that the improvement is predictable: each doubling of compute produces a predictable reduction in loss.

The paper reported specific exponents for each factor: the loss scaled with compute with an exponent of approximately 0.050, with parameters (model size) with an exponent of approximately 0.076, and with dataset size with an exponent of approximately 0.095. (Larger exponents mean faster loss reduction per unit increase, so data had the largest single-factor exponent — though this nuance would be the source of the later Chinchilla correction.)

This was a significant finding. Before the scaling laws, the AI research community did not have a clear theory of how model performance would improve with scale. Researchers knew that bigger models were generally better, but the relationship was not well understood. The Kaplan paper provided that understanding — and it suggested that the path to more capable AI was, at least in part, a matter of scaling up.

The paper had three key practical implications. First, it suggested that the most important input for AI capability was compute — the raw amount of computation used to train the model. Second, it suggested that the returns to scale — how output changes when all inputs are increased proportionally — were predictable, which meant that AI companies could plan their investments with some confidence. And third, it suggested that the path to more capable AI did not require fundamental breakthroughs — it required more of what was already working.

This last point was controversial. Some researchers argued that the scaling laws were an empirical observation, not a law of nature, and that they might not hold at larger scales. Others argued that the laws ignored important factors — like the quality of the training data, the architecture of the model, and the efficiency of the training algorithm. But the laws were, for the most part, accepted — and they drove the AI industry’s strategy for the next five years. (The Kaplan paper itself was never formally published in a peer-reviewed venue; it remained an arXiv preprint. Its influence came from being read and cited directly, not from conference proceedings.)

---

The Chinchilla Correction

In March 2022, two years after the Kaplan paper, researchers at DeepMind published a paper that significantly revised the scaling laws. The paper, “Training Compute-Optimal Large Language Models” (Hoffmann et al., arXiv:2203.15556, submitted March 31, 2022; published at NeurIPS 2022), introduced the “Chinchilla” model — a 70-billion-parameter model trained on 1.4 trillion tokens — and showed that Kaplan had under-weighted the importance of data.

Kaplan’s laws suggested that the optimal strategy was to increase model parameters faster than training data: specifically, Kaplan’s compute-optimal allocation gave parameters growing as compute to the 0.73 power (N ∝ C^0.73) and data growing as compute to the 0.27 power (D ∝ C^0.27) — meaning that for a fixed increase in compute, parameters should grow much faster than data. Chinchilla showed that the optimal strategy was to increase them at roughly the same rate — more data, smaller models.

This “compute-optimal scaling” correction was significant. It meant that AI companies that had followed Kaplan’s laws — building very large models trained on relatively small datasets — were not using their compute efficiently. Chinchilla suggested that they should be training smaller models on more data, which would produce better results for the same amount of compute.

The Chinchilla correction reset the AI industry’s training strategy. After Chinchilla, most frontier model developers adjusted their training to use more data relative to model size. The correction did not invalidate the scaling laws — it refined them — but it showed that the laws were not as simple as Kaplan had suggested, and that the details of the scaling relationship mattered for practical decisions.

---

What It Meant

The Kaplan scaling laws were, arguably, the most influential AI paper of the 2020s. They gave the AI industry a strategy — “just add compute” (an informal summary of the paper’s implication, not a phrase Kaplan himself used) — and they gave investors a thesis: that the companies with the most compute would build the best models, and that the gap between the leaders and the followers would widen over time. This thesis drove hundreds of billions of dollars of investment in AI chips (specialised semiconductors such as GPUs, designed for the kind of computation used in training and running neural networks), data centres (the large facilities that house the computing infrastructure), and model training. The frontier model developers — OpenAI, Google, Anthropic, Meta, and others — led this investment, and the AI chip market, dominated by NVIDIA, became one of the largest and most strategically important markets in the technology industry.

The laws also had a dark side. If progress in AI is primarily a matter of scaling, then the companies with the most resources — the most compute, the most data, the most capital — will inevitably win. This concentration of power was, for many observers, a troubling implication. It suggested that the AI industry would be dominated by a small number of well-funded companies, and that smaller labs and open-source efforts (projects, typically at smaller labs and universities, that build and release AI models openly) would be unable to compete.

The DeepSeek-R1 release on January 20, 2025 challenged this implication. DeepSeek, the Chinese AI company, released DeepSeek-R1, achieving frontier-level performance with a fraction of the compute used by the American frontier labs (see B85). The release suggested that clever engineering could substitute for raw compute, and that the scaling laws were not as rigid as had been assumed. The scaling laws were not wrong — bigger models trained on more data with more compute were still better — but the relationship between scale and performance was more nuanced than the “just add compute” strategy had suggested.

---

---

Was the kaplan scaling laws inevitable — the product of forces too large to redirect — or was it a series of choices, each of which could have gone differently? The answer matters, because it determines whether the future is something that happens to us or something we make.