BERT, October 2018: The Most-Cited NLP Model for Two Years

---

The Innovation: Bidirectional Understanding

GPT-1, released four months earlier, had used the decoder part of the Transformer — the part that generates text by predicting the next word. BERT used the encoder — the part that understands text by looking at all the words simultaneously.

The key innovation was bidirectional understanding. Traditional language models, including GPT, read text from left to right — they predict the next word based on the words that came before. This is like reading a sentence word by word, without knowing what comes next. BERT, by contrast, looked at the entire sentence at once — it could see both the words before and the words after any given position. This allowed BERT to understand context more deeply, because the meaning of a word often depends on the words that follow it. (Consider the word “bank” in “I went to the bank of the river” versus “I went to the bank to deposit a cheque” — the word after “bank” disambiguates the meaning.)

The bidirectional approach was, by most accounts, a significant improvement over the unidirectional approach for understanding tasks. The unidirectional approach (used by GPT) was fine for generation — you generate text word by word, left to right, so you only need to look at the words that came before. But for understanding — for tasks like classification, question answering, and sentiment analysis — the bidirectional approach was better, because it allowed the model to consider the full context of each word. BERT’s bidirectional approach was, in some ways, the key to its dramatic results on understanding benchmarks.

---

The Training Tasks

BERT achieved bidirectional understanding through a technique called masked language modelling. Instead of predicting the next word, BERT randomly masked some words in a sentence and tried to predict them based on the surrounding context. For example, given “The cat sat on the [MASK],” BERT would try to predict “mat” or “floor” or “couch” — using the context of the entire sentence, not just the words before the blank.

The masked language modelling approach was, by most accounts, a clever solution to a technical problem. The problem was that a bidirectional model — a model that could see both the words before and the words after a given position — could not be trained using the standard next-word prediction approach, because the model would “cheat” by looking at the word it was supposed to predict. The solution was to mask some words — to hide them from the model — and to train the model to predict them based on the surrounding context. The approach allowed the model to be trained bidirectionally, without the cheating problem, and it was, by most accounts, the key technical innovation that made BERT possible.

BERT was also pre-trained on a second task, next sentence prediction: given two sentences, the model had to decide whether the second sentence logically followed the first. For example, given “The cat sat on the mat” and “The dog ran in the yard,” the model would have to decide whether the second sentence followed from the first (it does not). The next sentence prediction task was designed to teach the model about the relationships between sentences — a skill that was important for tasks like question answering and natural language inference, where the model had to understand how different sentences related to each other.

Together, masked language modelling and next sentence prediction taught BERT to understand both the local context of words and the relationships between sentences. The combination was, by most accounts, what made BERT so effective on a wide range of understanding tasks.

---

The Results and Impact

BERT set new state-of-the-art results on 11 NLP benchmarks simultaneously — a feat that was unprecedented. The benchmarks included GLUE (General Language Understanding Evaluation, a collection of NLP tasks), SQuAD (Stanford Question Answering Dataset, questions about passages of text), and SWAG (Situations With Adversarial Generations, multiple-choice questions about everyday situations). The results were so dominant that BERT became the standard foundation for NLP research for the next two years.

The results were, by most accounts, remarkable. Before BERT, the state of the art on most NLP benchmarks was held by a variety of different models — each task had its own best model, and the models were often quite different from each other. BERT, by contrast, achieved state-of-the-art results on all 11 benchmarks with a single model — a model that was pre-trained once on unlabeled text and then fine-tuned on each task. The achievement was, in some ways, a demonstration of the power of the pre-training plus fine-tuning approach: a single pre-trained model, adapted to each task with a small amount of fine-tuning, could outperform models that had been specifically designed for each task.

The BERT paper won the Best Paper Award at NAACL 2019 — the 2019 conference of the North American chapter of the Association for Computational Linguistics (ACL), the premier professional society for computational linguistics and NLP. The award reflected the paper’s immediate and substantial impact on the field. The paper became one of the most-cited in computer science, and BERT-based models dominated NLP research until the rise of GPT-3 in 2020.

---

Google Search Integration

In October 2019, Google integrated BERT into its search engine — the first time a Transformer model was deployed at scale in a product used by billions of people. Google’s blog post announcing the rollout, “Understanding searches better than ever before,” was published on October 25, 2019. Google estimated that BERT affected about 10 percent of all English-language searches — roughly one in ten — improving the search engine’s ability to understand the meaning of queries, particularly longer, more conversational ones.

The integration was, by most accounts, a significant milestone for the field. BERT was the first Transformer model to be deployed in a product used by billions of people, and the deployment demonstrated that the technology was not just a research curiosity but a practical tool that could improve real-world products. The integration was also, in some ways, a validation of the pre-training plus fine-tuning approach — Google had decided that the approach was mature enough to deploy in its most important product, and the deployment was, by most accounts, successful.

BERT was eventually rolled out to more than 70 languages, beginning December 9, 2019. The multilingual rollout was, by most accounts, a significant engineering effort — BERT had to be adapted to work with languages that had different structures, different vocabularies, and different writing systems. But the rollout demonstrated that the approach was not limited to English, and it brought the benefits of BERT to users around the world.

---

BERT vs GPT: Two Paths

BERT and GPT represented two different philosophies of language modelling. BERT was an encoder — it was designed to understand text, not to generate it. GPT was a decoder — it was designed to generate text, not just to understand it. BERT was better at understanding tasks (classification, question answering, sentiment analysis — tasks that require understanding text rather than generating it). GPT was better at generation tasks (writing, translation, summarisation — tasks that require generating text rather than just understanding it).

For two years, BERT dominated NLP research. But the GPT line had a crucial advantage: it could generate text. And as GPT models got larger — GPT-2 with 1.5 billion parameters in 2019, GPT-3 with 175 billion in 2020 — the generation capability became increasingly powerful, and the GPT line eventually surpassed BERT in both understanding and generation. By 2022, the GPT line had produced ChatGPT, and the BERT line had been largely absorbed into the GPT ecosystem.

The competition between BERT and GPT was, in some ways, the defining competition of the 2018-2022 period in NLP. The two approaches represented two different theories of how language models should be built — the encoder approach, which prioritised understanding, and the decoder approach, which prioritised generation. The competition was, by most accounts, productive — it drove rapid progress in both approaches, and it gave the field a clear picture of the strengths and weaknesses of each. The resolution — the victory of the decoder approach — was not, in retrospect, inevitable. It was the result of specific technical developments (the discovery of in-context learning in GPT-3, the development of RLHF in InstructGPT) that made the generative approach more powerful than the understanding approach. Without these developments, BERT might have remained the dominant paradigm, and the field of NLP might have developed quite differently.

---

The BERT Ecosystem

BERT’s influence extended well beyond the original model. The BERT paper spawned a whole ecosystem of variants and extensions, each addressing a limitation of the original or extending it to a new domain:

RoBERTa (Robustly Optimized BERT Pretraining Approach, Liu et al., Facebook AI, July 2019) improved on BERT by training longer, on more data, with larger batch sizes, and by removing the next sentence prediction task (which the authors found did not help). RoBERTa demonstrated that BERT’s original training had been suboptimal, and that with better training, the same architecture could achieve significantly better results.

DistilBERT (Sanh et al., Hugging Face, October 2019) was a smaller, faster, cheaper version of BERT, produced by knowledge distillation — a technique that trained a smaller model to mimic the behaviour of a larger one. DistilBERT was about 40 percent smaller than BERT and about 60 percent faster, while retaining most of BERT’s performance. The model was designed for deployment in environments where the full BERT was too expensive — mobile devices, edge computing, real-time applications.

ALBERT (A Lite BERT, Lan et al., Google, September 2019) reduced the memory consumption of BERT by sharing parameters across layers, allowing larger models to be trained with the same memory budget. ALBERT also replaced the next sentence prediction task with a sentence ordering prediction task, which the authors found more effective.

The BERT ecosystem also included Hugging Face Transformers — the open-source library that provided pre-trained implementations of BERT and hundreds of other Transformer models. The library, originally released as pytorch-pretrained-bert in late 2018 (see B34 — PyTorch), became the standard tool for working with pre-trained language models, and it played a crucial role in making BERT and its variants accessible to researchers and developers around the world.

---

The Legacy

But BERT’s influence remains. The bidirectional understanding that BERT pioneered is still used in many NLP systems, and the pre-training paradigm — first pre-train on a large amount of unlabeled text, then fine-tune on specific tasks — that BERT and GPT-1 both demonstrated has become the standard approach for all of NLP. BERT was, for two years, the most important model in the field — and it laid the groundwork for everything that followed.

The legacy is also, in some ways, the legacy of a particular approach to AI research. BERT was, by most accounts, an engineering achievement rather than a scientific breakthrough. The individual components — the Transformer architecture, the pre-training plus fine-tuning recipe, the masked language modelling task — were not new. What was new was the combination: the specific way that BERT put these components together, trained them at scale, and demonstrated that the result could achieve state-of-the-art performance on a wide range of tasks. The achievement was, in some ways, a reminder that progress in AI often comes not from single breakthroughs but from the careful combination of existing ideas, executed well and at scale. BERT was, in this sense, a model of how AI research works — not through single eureka moments, but through the patient, careful work of combining, testing, and scaling.

---

---

---

The legacy of BERT is still being written. The decisions made today — by engineers, by policymakers, by users — will determine whether it is remembered as a turning point or a footnote.