GPT-3, June 2020: The Pivot Point of the LLM Era

GPT-3

Paper date:: May 28, 2020 (arXiv:2005.14165) · API beta: June 11, 2020
Lab/Organisation:: OpenAI
Paper/Outcome:: “Language Models are Few-Shot Learners” (Brown et al., 31 authors)
Significance:: 175B-parameter model that demonstrated in-context learning; pivot point of the LLM era

GPT-3

A 175B-parameter decoder-only Transformer released by OpenAI in June 2020. GPT-3 demonstrated in-context learning — the ability to perform new tasks from prompt examples without weight updates — and pivoted the AI industry toward foundation models.

The Model

GPT-3 was, at the time of its release, the largest language model ever built. It had 175 billion parameters, trained on a corpus whose combined weighted dataset totaled about 499 billion tokens (the units of text that language models process, roughly equivalent to words or parts of words) — drawn from Common Crawl (410 billion tokens), WebText2 (19 billion), Books1 (12 billion), Books2 (55 billion), and Wikipedia (3 billion). The model was trained for a total of 300 billion tokens (less than the full pool, because the training process samples from the weighted mix). The model used the same Transformer decoder architecture as GPT-1 and GPT-2, but at a scale that was unprecedented.

The scale was the point. GPT-3 was the empirical confirmation of the Kaplan scaling laws (see B73), which had been published just five months earlier. Kaplan had predicted that bigger models would be better, and GPT-3 showed that the prediction was correct. The model’s performance on a wide range of NLP benchmarks was dramatically better than any previous model — not because of a new architecture or a new training technique, but because of scale.

The Discovery: In-Context Learning

The most significant finding of the GPT-3 paper was not the model’s benchmark performance. It was the discovery of in-context learning — the ability of the model to perform new tasks, without any additional training, simply by being given a few examples in the prompt.

Before GPT-3, if you wanted a language model to perform a new task — say, translating English to French — you needed to fine-tune the model on a dataset of English-French translation pairs. This required collecting labelled data, running a training process, and deploying the fine-tuned model. It was expensive and time-consuming.

GPT-3 changed this. With GPT-3, you could simply give the model a few examples of English-French translations in the prompt — “The cat sat on the mat → Le chat s’est assis sur le tapis” — and the model would generalise from those examples to translate new sentences. No fine-tuning required. No additional training. Just examples in the prompt.

Why “few-shot learning” mattered

The technical name for what GPT-3 did was few-shot learning — giving the model a few examples in the prompt to teach it a new task. The reason this mattered is economic, not just technical. Before GPT-3, every new task required its own expensive fine-tuning run: collect labelled data, train, deploy. After GPT-3, one model could do thousands of tasks, and “programming” it just meant writing a different prompt. The cost of building an AI application collapsed from “train a model” to “write a paragraph.” That collapse is what made the generative-AI startup wave of 2021–2023 possible.

This was called “few-shot learning,” and it was a transformative discovery. It meant that a single, large, pre-trained model could perform thousands of different tasks, without any task-specific training. The model was not specialised for any particular task; it was a general-purpose system that could be adapted to any task through the prompt. This capability — a single model serving as a foundation for many different applications — would later be termed “foundation model” by Stanford’s Center for Research on Foundation Models (CRFM) in August 2021, in the report “On the Opportunities and Risks of Foundation Models” (arXiv:2108.07258, 16 August 2021, 114 authors).

The API-Only Release

GPT-3 was not released as open weights. It was available only through OpenAI’s API, which opened in closed beta on June 11, 2020, with a waitlist that limited initial access. Unrestricted API access would not come until November 18, 2021. The API-only release was a significant commercial decision. It meant that OpenAI controlled who could use the model, what they could use it for, and how much they paid. It also meant that OpenAI could monetise the model directly, rather than relying on partnerships or licensing.

The API-only release was controversial. Researchers who wanted to study the model could not download it and run experiments on their own hardware. They could only access it through the API, which limited the kinds of experiments they could do and imposed costs that made large-scale research difficult. The open-source community, which had been able to build on GPT-2 after its full release, was shut out.

On September 22, 2020, Microsoft announced that it had licensed GPT-3 exclusively — meaning that while OpenAI continued to operate the public API, Microsoft was the only other company that could use GPT-3’s underlying model directly. The deal was a watershed moment in the OpenAI–Microsoft partnership and a clear signal of the commercial value that GPT-3 represented.

The decision to release GPT-3 through an API was, in retrospect, the moment when OpenAI completed its transformation from an open research lab to a closed commercial company. The “too dangerous to release” framing of GPT-2 had been the first step; the API-only release of GPT-3 was the second. By the time ChatGPT was released on November 30, 2022, the transformation was complete.

What It Meant

GPT-3 was the pivot point of the LLM era. It demonstrated, for the first time, that a single large language model could perform a wide range of tasks without task-specific training. It showed that scale — more parameters, more data, more compute — was a viable path to more capable AI. And it established the commercial model — API access, closed weights, pay-per-use — that would dominate the AI industry for the next five years.

The model also intensified the AI race. Google, Meta, Anthropic, and other companies began building their own large language models, trying to match or exceed GPT-3’s capabilities. Anthropic (the AI company founded in 2021 by former OpenAI researchers, led by siblings Dario Amodei and Daniela Amodei — Dario had been a co-author of the GPT-2 paper) began building the Claude models as a competitor to GPT. Meta later released its LLaMA models as open weights — a deliberate counter-strategy to OpenAI’s closed approach. The scaling laws suggested that the path to better models was more scale, and the race to build the biggest models was on.

GPT-3 did not, by itself, produce the AI revolution. The model was impressive, but it was not yet useful enough for mainstream consumers. That would come with InstructGPT (released January 27, 2022; see B75), which used RLHF (Reinforcement Learning from Human Feedback) to make the model follow instructions, and with ChatGPT (released November 30, 2022), which brought the technology to the mass market. But GPT-3 was the foundation — the model that proved the concept, established the commercial model, and set the race in motion.