GPT-1, June 2018: The Start of the GPT Line

---

The Context

The context for GPT-1 was a field in transition. Natural language processing had, for most of its history, been dominated by supervised learning — the practice of training models on labelled data, where the model was given examples of inputs and their correct outputs. The approach had been successful for many tasks — sentiment analysis, spam detection, named entity recognition — but it had a fundamental limitation: it required large amounts of labelled data, which was expensive and time-consuming to create. For most NLP tasks, the labelled datasets were small, and the models that could be trained on them were limited by the size of the data.

In 2017 and early 2018, several research groups began exploring an alternative approach: unsupervised pre-training. The idea was to train a model on a large amount of unlabeled text — text without any labels, which was much easier to obtain — and then to fine-tune the pre-trained model on specific tasks using smaller amounts of labelled data. The approach had been explored in various forms for years, but it had not, until 2017-2018, been demonstrated to work at scale.

Two papers, in particular, set the stage for GPT-1. The first was the Transformer paper — “Attention Is All You Need” (Vaswani et al., June 2017, arXiv:1706.03762) — which introduced the Transformer architecture. The Transformer was a neural network design that used self-attention — a mechanism that allowed each word in a sentence to attend to every other word — to process text. The architecture was, by most accounts, more powerful and more parallelisable than the recurrent neural networks that had dominated NLP, and it would become the foundation for virtually all subsequent language models.

The second was the ULMFiT paper — “Universal Language Model Fine-tuning for Text Classification” (Howard & Ruder, January 2018, arXiv:1801.06146) — which demonstrated that the pre-training plus fine-tuning approach could work for NLP. ULMFiT used a recurrent neural network (an LSTM) rather than a Transformer, but it established the general recipe: pre-train on unlabeled text, then fine-tune on specific tasks. The paper was, by most accounts, influential in convincing the NLP community that the approach was viable.

GPT-1 was, in some ways, the combination of these two ideas — the Transformer architecture and the pre-training plus fine-tuning recipe — applied at a scale that had not been attempted before.

---

The Idea

The idea behind GPT-1 was not new. It combined two existing concepts: the Transformer architecture, which had been introduced by Google researchers in June 2017 (Vaswani et al., arXiv:1706.03762), and the practice of pre-training — training a model on one task before fine-tuning it on another.

What was new was the specific combination. Previous approaches to natural language processing (NLP) — the field of AI concerned with understanding and generating human language — had typically trained models from scratch on each specific task: sentiment analysis (positive or negative), question answering, text classification, and so on. Each task required a separate model, trained on labelled data — text annotated with the correct answer by human annotators, which is expensive to create. Most NLP tasks had relatively small labelled datasets, and the field was, in effect, bottlenecked by the cost of human annotation.

GPT-1 took a different approach. Instead of training from scratch on each task, it first trained the model on a large amount of unlabeled text — text without human annotations, which is much easier to obtain — from a dataset called BooksCorpus, a collection of about 7,000 unique unpublished books spanning a variety of genres.

This “pre-training” taught the model the general patterns of language: grammar, vocabulary, common-sense reasoning, and basic world knowledge. Then, for each specific task, the model was “fine-tuned” — slightly adjusted using a small amount of labelled data for that task.

The advantage of this approach was that the pre-training could be done once, on a large amount of freely available unlabeled text, and the resulting pre-trained model could then be adapted to many different tasks with relatively small amounts of labelled data. This was more efficient than training from scratch, and it often produced better results, because the model had already learned general language patterns that were useful across tasks.

---

The Architecture

GPT-1 used the decoder part of the Transformer architecture — the part that generates text by predicting the next word in a sequence. (The original Transformer, introduced by Google, had both an encoder (designed to understand input text) and a decoder (designed to generate output text). GPT used only the decoder.) The model had 12 layers, 768-dimensional hidden states, 12 attention heads, and 117 million parameters, and was trained on the BooksCorpus dataset.

The choice of the decoder architecture was significant. It meant that GPT was, at its core, a next-word prediction model — it predicted what word came next in a sequence. This was different from BERT, which Google would release four months later (see B71) and which used the encoder architecture. The difference would define the two main branches of Transformer-based language models: the generative branch (GPT, which generates text) and the understanding branch (BERT, which understands text).

The decision to use the decoder, rather than the encoder or the full encoder-decoder, was, by most accounts, a deliberate strategic choice. OpenAI was, from the beginning, interested in building generative models — models that could generate text, not just understand it. The decoder architecture was the natural choice for a generative model, because the decoder was designed to generate sequences. The choice would, over the following years, prove to be consequential — the GPT line, built on the decoder, would eventually surpass the BERT line, built on the encoder, in both understanding and generation, and the generative approach would become the dominant paradigm in NLP.

---

The Results

GPT-1 achieved state-of-the-art (SOTA) results — the best performance reported on a benchmark — on several NLP benchmarks, including natural language inference (determining the relationship between two sentences), question answering (answering questions about a passage of text), and text classification (assigning a category to a piece of text, such as positive or negative sentiment). The paper reported absolute improvements of 8.9% on commonsense reasoning (Stories Cloze Test), 5.7% on question answering (RACE), and 1.5% on textual entailment (MultiNLI). The results were not dramatically better than existing models, but they demonstrated that the pre-training plus fine-tuning approach was viable. The model was particularly strong on tasks that required understanding long passages of text, which was likely because the BooksCorpus training data consisted of full books.

The results were, by most accounts, solid but not spectacular. GPT-1 was a competent model, but it was not a breakthrough — it was a proof of concept, a demonstration that the approach worked. The improvements over existing models were real, but they were not the dramatic improvements that would come with GPT-2 (1.5 billion parameters, 2019) and GPT-3 (175 billion parameters, 2020). GPT-1 was, in some ways, the quiet beginning of a revolution — the model that established the approach, but not the model that made the world notice.

---

The Reception

The paper was influential, but it did not attract widespread attention outside the NLP research community. GPT-1 was a competent model, but it was not a breakthrough. It was a proof of concept — a demonstration that the approach worked — rather than a finished product. The breakthroughs would come later, with GPT-2 in 2019 (1.5 billion parameters) and GPT-3 in 2020 (175 billion parameters).

The relatively quiet reception was, in some ways, a measure of the model’s position in the GPT line. GPT-1 was the first model in the line, and it was, by most accounts, the least impressive. The model’s capabilities were limited — it could not generate the fluent, coherent text that GPT-2 would produce, and it could not perform the wide range of tasks that GPT-3 would handle. The model was, in some ways, a research artefact — a demonstration that the approach was viable, but not a model that anyone would use in production.

But the paper was, within the NLP research community, widely read and widely cited. The approach — pre-training on unlabeled text, then fine-tuning on specific tasks — was, by most accounts, the right approach, and the paper was one of the first to demonstrate it at scale. The approach would, over the following years, become the standard approach for NLP, and GPT-1 was one of the models that established it.

---

The Legacy

But GPT-1 was the start of something. It established the GPT line — the series of models that would, within five years, produce ChatGPT and transform the technology industry. It introduced the pre-training plus fine-tuning paradigm that would become the standard approach for NLP. And it planted the seed of the idea that would, in GPT-3, become one of the most important discoveries in AI history: that a model trained only to predict the next word could, if made large enough, learn to perform a wide range of tasks without any task-specific training.

The legacy is also, in some ways, the legacy of a strategic choice. OpenAI’s decision to use the decoder architecture — the generative branch of the Transformer — was, in 2018, a bet. The encoder branch (which BERT would popularise) was, at the time, the more obvious choice for NLP understanding tasks, and the generative approach was, by most accounts, less proven. OpenAI’s bet on the decoder was, in some ways, a bet on the future — a bet that generative models, which could produce text, would eventually surpass understanding models, which could only classify it. The bet would, over the following years, prove to be correct. The GPT line, built on the decoder, would eventually surpass the BERT line, built on the encoder, and the generative approach would become the dominant paradigm in NLP. GPT-1 was the first step in that journey — the model that established the approach, and the model that set the line on its course.

---

---

---

Was gpt-1 inevitable — the product of forces too large to redirect — or was it a series of choices, each of which could have gone differently? The answer matters, because it determines whether the future is something that happens to us or something we make.