Chinchilla, March 2022: The Compute-Optimal Reset

Chinchilla

Date:: March 31, 2022 (arXiv) · Published at NeurIPS 2022
Lab/Organisation:: DeepMind
Paper/Outcome:: “Training Compute-Optimal Large Language Models” (Hoffmann, Borgeaud, Mensch et al., 22 authors; arXiv:2203.15556)
Significance:: Reversed Kaplan’s recipe — more data, smaller models — and reset training strategy

March 31 2022 A team of

March 31, 2022. A team of researchers at DeepMind — Google’s London AI lab — posts a paper to arXiv titled “Training Compute-Optimal Large Language Models.” The paper, twenty-two authors long, is led by Jordan Hoffmann, Sebastian Borgeaud, and Arthur Mensch. (Mensch would later leave DeepMind to co-found Mistral AI, one of Europe’s leading open-weight AI companies — a useful reminder that the people who wrote the Chinchilla recipe went on to apply it elsewhere.) The paper describes a series of experiments — over four hundred model training runs, at scales ranging from 70 million to over 16 billion parameters — designed to answer a simple but crucial question: given a fixed amount of compute, how should you allocate it between model size and training data? The answer, the paper argues, is not what the field has been doing. The standard practice, established by OpenAI’s Kaplan scaling laws (see B73) in January 2020, has been to scale up model size much faster than data size — to build bigger models, trained on roughly the same amount of data.

The DeepMind paper argues that this is wrong. The optimal strategy, the paper shows, is to scale model size and data size roughly equally — to train smaller models on much more data. The paper introduces a new model, Chinchilla, that demonstrates the principle: a 70-billion-parameter model, trained on 1.4 trillion tokens, that outperforms GPT-3 (175 billion parameters) and Gopher (280 billion parameters) despite being much smaller. The paper is, by most accounts, the most important scaling-law paper since Kaplan, and it resets the field’s training strategy. Within a year, every major AI lab will have adopted the Chinchilla recipe.

Chinchilla Scaling Laws

Compute-optimal scaling relationships published by DeepMind in March 2022 (Hoffmann et al.). Chinchilla showed that prior models (including GPT-3) were undertrained for their size — the optimal strategy is to scale data and parameters equally — and reset training-recipe norms across the industry.

The Kaplan Paradigm

Chinchilla scaling laws are the compute-optimal scaling relationships DeepMind published in March 2022 (Hoffmann et al.). The paper showed that prior frontier models — including GPT-3 — were significantly undertrained for their parameter count. The optimal strategy is to scale data and parameters equally, doubling data for every doubling of parameters.

To understand why Chinchilla mattered, you have to understand the paradigm it overturned. In January 2020, Jared Kaplan and his colleagues at OpenAI published a paper — “Scaling Laws for Neural Language Models” (see B73) — that gave the field its central prediction: the performance of a language model depends, in a predictable way, on three factors — the size of the model, the amount of data it is trained on, and the amount of compute used for training. The relationship was, Kaplan showed, a power law — each factor, increased independently, produced a predictable improvement in performance, and the improvement continued smoothly across many orders of magnitude.

The Kaplan scaling laws had a profound influence on the field. They provided, for the first time, a quantitative basis for the strategy that OpenAI had been pursuing — the strategy of building ever-larger models, trained on ever-more data, with ever-more compute. The laws said, in essence, that the path to more capable AI was to scale up — that if you built a bigger model and trained it on more data, you would get a better system, and that this would continue to hold as you scaled. The prediction was, for OpenAI, enormously encouraging. It suggested that the path to AGI — or at least to dramatically more capable AI — was not a matter of fundamental breakthroughs but of engineering: build bigger models, get more data, spend more compute.

But the Kaplan laws had a specific implication that, in retrospect, turned out to be misleading. The laws suggested that, for a fixed amount of compute, you should allocate most of your budget to model size. Kaplan’s compute-optimal allocation gave parameters growing as compute to the 0.73 power (N ∝ C^0.73) and data growing as compute to the 0.27 power (D ∝ C^0.27) — a ratio of roughly 2.7:1, often loosely summarised as “3:1.” In other words, you should build bigger models, but you should not increase the training data by nearly as much. The implication was that model size was the primary driver of performance, and that data was, relatively speaking, less important.

This implication shaped the field’s practice for the next two years. GPT-3, released in June 2020 (see B74), was a 175-billion-parameter model — vastly larger than GPT-2 (1.5 billion parameters) — but it was trained on roughly 300 billion tokens, only modestly more than GPT-2’s ~40 billion. The same pattern held for other large models: Gopher (DeepMind, 280 billion parameters, ~300 billion tokens), Jurassic-1 (AI21 Labs, 178 billion), Megatron-Turing NLG (NVIDIA/Microsoft, 530 billion). All of these models were big — much bigger than GPT-3 — but none of them were trained on dramatically more data. The field was, in essence, following the Kaplan recipe: build bigger models, train on roughly the same data.

The Chinchilla Experiments

The DeepMind team, led by Hoffmann, Borgeaud, and Mensch, decided to test the Kaplan recipe. The test was straightforward in principle but expensive in practice: train a large number of models, at different scales, with different ratios of model size to data size, and measure which combination produced the best performance for a given amount of compute. The team ran over four hundred training runs, at scales ranging from 70 million to over 16 billion parameters, and they measured the performance of each model on a standard set of benchmarks.

The results were striking. The Kaplan recipe, the team found, was wrong. The optimal strategy was not to scale model size faster than data size. The optimal strategy was to scale them roughly equally — for every doubling of compute, you should increase both model size and data size by about the same factor. The implication was that the field had been building models that were too big, trained on too little data. The right strategy was to build smaller models, trained on much more data.

Why the Chinchilla correction mattered commercially

Kaplan’s prescription — spend on parameters, not data — had led frontier labs to build very large models trained on relatively small datasets. Chinchilla showed that this was inefficient: for the same compute budget, you could get a better model by training a smaller model on more data. The implication was that the data pipeline — how much text you could scrape, clean, and feed into training — mattered as much as the chip budget. After Chinchilla, frontier labs scrambled to acquire more training data, which in turn drove the wave of licensing deals with publishers, the scraping of Reddit and other platforms, and the lawsuits over training-data provenance that would dominate the mid-2020s.

The discrepancy was substantial. For a given compute budget, the Chinchilla-optimal model was, depending on the scale, 3 to 10 times smaller than the Kaplan-optimal model, and it was trained on 3 to 10 times more data. The practical implication was clear: the field had been misallocating its compute. The billions of dollars that had been spent training enormous models on relatively modest datasets would, if the Chinchilla recipe had been followed, have produced better models — smaller, cheaper to run, and more capable.

The Chinchilla Model

To demonstrate the principle, the DeepMind team trained a model called Chinchilla. Chinchilla had 70 billion parameters — less than half the size of GPT-3 (175 billion) and about a quarter the size of Gopher (280 billion). But Chinchilla was trained on 1.4 trillion tokens — roughly 4.7 times more data than GPT-3 (~300 billion tokens) and 4.7 times more than Gopher (also ~300 billion). The total compute budget for Chinchilla was the same as for Gopher — both models were trained with the same amount of computational effort. The difference was entirely in the allocation: Chinchilla spent its compute on a smaller model with more data, while Gopher spent its compute on a bigger model with less data.

The results were decisive. Chinchilla outperformed Gopher on every benchmark. On MMLU (Massive Multitask Language Understanding) — the standard benchmark that tests a model’s knowledge across fifty-seven subjects, from history to medicine to mathematics — Chinchilla scored 67.5% (67.6% in the appendix table), about 7.5 percentage points higher than Gopher’s 60.0%. Chinchilla also outperformed GPT-3 (which scored 43.9% on MMLU) and the much larger Megatron-Turing NLG (530 billion parameters, which scored around 50%). The implication was clear: a 70-billion-parameter model, trained on enough data, could outperform models that were 2 to 7 times larger. The Kaplan recipe — build bigger models — had been decisively overturned.

The Chinchilla result was, for the field, a wake-up call. The standard practice — building ever-larger models on roughly the same data — had been shown to be suboptimal. The new practice — building smaller models on much more data — was, the Chinchilla paper demonstrated, both cheaper and more effective. The result was, in some ways, embarrassing for the labs that had followed the Kaplan recipe. The billions of dollars spent on Gopher, on Jurassic-1, on Megatron-Turing NLG had, by the Chinchilla analysis, been largely wasted — the same compute, allocated differently, would have produced better models.

The Impact

The impact of the Chinchilla paper was immediate and broad. Within months of its publication, every major AI lab had adjusted its training strategy. The new models, trained in 2022 and 2023, followed the Chinchilla recipe: they were smaller, and they were trained on more data. Meta’s LLaMA series, released in February 2023, was the most prominent example. LLaMA was a 7-billion to 65-billion-parameter model (depending on the variant), trained on 1.0 to 1.4 trillion tokens — roughly the Chinchilla recipe. LLaMA, despite being much smaller than GPT-3, matched or exceeded its performance, demonstrating that the Chinchilla approach worked in practice as well as in theory.

The Chinchilla paper also raised a concern that would, in the following years, become central to the field: the “data wall.”

The “data wall” — Chinchilla’s unintended consequence

The Chinchilla recipe said that you should train on more data — much more data — than the field had been using. But the amount of high-quality text available for training was finite. The internet, which had been the primary source of training data, contained perhaps a few trillion tokens of usable text. A Chinchilla-optimal 1-trillion-parameter model would require approximately 20 trillion tokens — far more than the available supply. The concern was that the field, by following the Chinchilla recipe, would soon run out of data, and that the scaling laws — the laws that had driven the field’s progress — would break down when the data ran out. As of 2026, AI labs have responded by developing new sources of training data — synthetic data (text generated by AI models themselves), multimodal data (images, video, audio alongside text), and proprietary data (licensed from publishers, libraries, and other content owners). The question of whether these new sources will be sufficient to sustain the scaling laws, or whether the field will hit a wall, remains open. But the question itself — the recognition that data, not just compute, is a binding constraint — is a direct consequence of the Chinchilla paper.

The Legacy

The legacy of the Chinchilla paper is, by any measure, the legacy of a revised understanding of how to train large language models. The Kaplan scaling laws, which had guided the field from January 2020 to March 2022, were overturned. The new recipe — smaller models, more data — became the standard, and it has, in the years since, produced a generation of models that are both more capable and more efficient than the models that preceded them. The practical impact — the billions of dollars saved, the improved performance, the democratisation of the technology (smaller models are cheaper to run and easier to deploy) — is substantial.

The legacy is also, in important ways, the legacy of a methodological lesson. The Kaplan scaling laws, despite their influence, had been based on a relatively small number of training runs, and they had not been adequately tested at scale. The Chinchilla paper, with its four-hundred-plus training runs and its careful experimental design, demonstrated that the Kaplan laws were wrong — and it demonstrated this not by argument but by experiment. The lesson — that scaling laws need to be empirically validated, not just assumed — is one that the field has taken to heart. Subsequent scaling-law research has been more careful, more empirical, and more rigorous.

The legacy is also, finally, a reminder of the importance of questioning assumptions. The Kaplan recipe — build bigger models — had become, by 2022, the conventional wisdom of the field. It was followed by every major lab, it was the basis for billions of dollars of investment, and it was, by most accounts, simply assumed to be correct. The Chinchilla team, by questioning the assumption and by running the experiments that the assumption demanded, showed that the conventional wisdom was wrong. The lesson — that even the most widely held assumptions should be tested, and that the willingness to test them is one of the most important virtues of a research community — is one that the field of AI, which is prone to both hype and groupthink, would do well to remember. The Chinchilla paper was, in the best sense, a piece of corrective science — a demonstration that the field had been going in the wrong direction, and a redirection toward the right one.