InstructGPT, January 2022: RLHF Becomes Production AI

---

The Problem

GPT-3 (see B74) was a remarkable model, but it had a significant limitation: it did not follow instructions. If you asked GPT-3 to “write a poem about a cat,” it might write a poem about a cat — or it might write about something else entirely. The model was trained to predict the next word in a sequence, not to follow instructions. It would generate whatever text was most statistically likely given the prompt, which was not always what the user wanted.

This made GPT-3 difficult to use. Users had to carefully craft their prompts to get the model to produce the desired output — the early art of prompt engineering (see B88). But even with careful prompting, the model would sometimes produce unhelpful, irrelevant, or inappropriate responses. It would generate text that was fluent and grammatically correct but not useful. A user who asked GPT-3 a question might get a confident-sounding answer that was completely wrong. A user who asked GPT-3 to do something harmful might get a willing response. A user who asked GPT-3 to admit its limitations might get a response that doubled down on a false claim. The model was, in these respects, not aligned with what users wanted.

OpenAI wanted to fix this. They wanted a model that would follow instructions — that would do what the user asked, not just generate statistically likely text. And they wanted the model to be safe — to refuse harmful requests, to avoid producing toxic content, and to be honest about its limitations. The goal was, in some ways, straightforward: take the capable but unaligned GPT-3, and make it aligned — make it do what users wanted, safely and helpfully. The challenge was figuring out how.

---

The Solution: RLHF

The solution was Reinforcement Learning from Human Feedback, or RLHF — a training technique that uses human feedback to teach a language model what kinds of responses are good and what kinds are bad. RLHF was not invented by OpenAI; the foundational paper was Christiano, Leike, Amodei et al., 2017 (“Deep reinforcement learning from human preferences,” arXiv:1706.03741), and earlier language-model applications existed (Stiennon et al. 2020 on summarisation). But InstructGPT was the most prominent large-scale application of RLHF to a production language model, and the paper that made RLHF the default alignment technique in the industry. The process has three steps:

The result is a model that follows instructions, produces helpful responses, and avoids harmful content — not because it understands instructions or safety, but because it has been trained to produce responses that human raters prefer. The technique is, in some ways, a clever way of aligning a model with human preferences without needing to specify those preferences in advance. Instead of writing rules about what the model should and should not do, OpenAI simply showed the model examples of good and bad responses, and let the model learn the patterns.

The three-step process was, by most accounts, more effective than any previous approach to alignment. Step 1 (supervised fine-tuning) gave the model a basic understanding of what a good response looked like. Step 2 (reward model) gave the model a way to evaluate the quality of any response. And step 3 (reinforcement learning) allowed the model to improve, over many iterations, by learning from the reward model’s feedback. The combination was, in some ways, the key — each step addressed a limitation of the previous steps, and the result was a model that was significantly more aligned than the original GPT-3.

---

The Results

The paper reported that the 175-billion-parameter InstructGPT was preferred over the 175-billion-parameter GPT-3 by human raters 85% of the time (zero-shot) and 71% of the time (few-shot) on instruction-following tasks. The model was also better on the three qualities that OpenAI had identified as essential for a safe and useful AI system — helpful, honest, and harmless (HHH) — with truthfulness measured as a proxy for honesty.

The results were, by most accounts, dramatic. The InstructGPT model was the same size as GPT-3 — it had the same number of parameters, was trained on the same base data, and used the same architecture. The only difference was the RLHF training. And yet, the InstructGPT model was preferred over GPT-3 by 85 percent of human raters. The result was, in some ways, a demonstration of the power of alignment — the same model, trained to follow instructions and to produce responses that humans preferred, was dramatically more useful than the original.

The results also showed that RLHF did not come at the cost of capability. One concern with alignment techniques was that they might make models less capable — that a model trained to be safe and helpful might be less able to perform the tasks it was originally trained for. The InstructGPT results showed that this concern was, in most cases, unfounded. The InstructGPT model was, on most capability benchmarks, as good as or better than GPT-3, and it was significantly better on the alignment benchmarks. The result was, in some ways, the best of both worlds — a model that was both more capable and more aligned.

---

Why It Mattered

InstructGPT was the breakthrough that made ChatGPT possible. Without RLHF, ChatGPT would have been GPT-3 with a chat interface — a model that could generate text but could not follow instructions, could not refuse harmful requests, and could not produce consistently helpful responses. RLHF was the technique that transformed GPT-3 from a research model into a useful product.

The connection between InstructGPT and ChatGPT was, by most accounts, direct. ChatGPT, released on November 30, 2022, was built on the same technology as InstructGPT — it was, in OpenAI’s phrasing, a “sibling model” to InstructGPT. The ChatGPT model (GPT-3.5) was trained using the same RLHF process, and it exhibited the same instruction-following, helpful, and safety-oriented behaviour. The difference was the interface — InstructGPT was accessed through an API, while ChatGPT was accessed through a chat interface. But the underlying technology was the same, and the breakthrough — the RLHF training that made the model useful — was the same.

The InstructGPT paper was, in this sense, the technical foundation of ChatGPT. Without the InstructGPT work, ChatGPT would not have been possible. The paper demonstrated that RLHF could transform a language model from a research artefact into a useful product, and it provided the specific techniques and the specific results that made the transformation possible. The fact that ChatGPT, rather than InstructGPT, became the public face of the technology was, in some ways, a matter of timing and presentation — ChatGPT had a better interface, a better launch, and a better name. But the technical breakthrough was InstructGPT.

---

The HHH Triad

The triad came from Askell et al. 2021, and it would become the standard framing for AI alignment in the post-InstructGPT era. The three qualities — helpful, honest, and harmless — were, by most accounts, a useful way of thinking about what it meant for a model to be aligned:

• Helpful: the model should do what the user asked, and should do it well. It should produce responses that are relevant, accurate, and useful.
• Honest: the model should tell the truth. It should not make false claims, it should not be deceptive, and it should admit when it does not know something.
• Harmless: the model should not produce harmful content. It should refuse requests that could lead to harm, it should avoid producing toxic or offensive content, and it should not assist with dangerous activities.

The HHH triad was, in some ways, a simplification of a complex problem. The three qualities could, in some cases, conflict — a model that was maximally helpful might, in some cases, produce harmful content (if the user asked for something harmful). A model that was maximally harmless might, in some cases, be unhelpful (if it refused requests that were not actually harmful). The tensions between the three qualities were, and remain, one of the central challenges of alignment. But the triad provided a useful framework for thinking about the problem, and it gave the industry a shared vocabulary for discussing it.

---

The Industry Adoption

RLHF also became the dominant alignment technique — the dominant method for ensuring AI systems behave in accordance with human intentions — in the AI industry. After InstructGPT, virtually every frontier model developer adopted some form of RLHF. Anthropic developed Constitutional AI (arXiv:2212.08073, December 2022) — a variant that replaced human feedback with AI feedback (sometimes called RLAIF). Google, Meta, and others developed their own variants. The technique became so widespread that “RLHF” entered the mainstream technology vocabulary — a term that was once known only to AI researchers became part of the industry’s common language.

The adoption was, by most accounts, rapid and comprehensive. Within a year of the InstructGPT paper, every major frontier lab had implemented some form of RLHF, and the technique had become the standard approach to alignment. The adoption was, in some ways, a measure of the paper’s importance — the technique was so clearly effective that the entire industry adopted it, and adopted it quickly. The adoption was also, in some ways, a measure of the field’s maturity — the technique was well-documented, well-tested, and well-understood, and the field was able to implement it rapidly and at scale.

The industry adoption also drove a wave of innovation in alignment techniques. After InstructGPT, researchers explored a range of variations — RLAIF (which replaced human feedback with AI feedback), DPO (Direct Preference Optimization, which simplified the RLHF process), and a variety of other approaches. The innovation was, in some ways, a sign of the field’s health — the InstructGPT paper had established the basic approach, and the field was now exploring variations and improvements. The result was a rapid progress in alignment techniques, driven by the competition between the frontier labs and by the open sharing of techniques through papers and open-source releases.

---

The Legacy

InstructGPT’s legacy is, by any measure, the legacy of ChatGPT. The InstructGPT paper was the technical foundation of ChatGPT, and ChatGPT was, by most accounts, the most consequential AI product of the 2020s. The paper demonstrated that RLHF could transform a language model from a research artefact into a useful product, and it provided the specific techniques and the specific results that made the transformation possible. Without InstructGPT, there would have been no ChatGPT. Without ChatGPT, the AI industry of 2023-2026 — the hundreds of billions of dollars of investment, the thousands of start-ups, the regulatory debates, the cultural conversation — would not have happened in the way it did.

The legacy is also, in some ways, the legacy of alignment as a practical discipline. Before InstructGPT, alignment was, by most accounts, a theoretical concern — a topic of academic research, but not a practical technique that companies used to build products. After InstructGPT, alignment was a practical discipline — a set of techniques that every frontier lab used, that every model card reported on, and that every product launch discussed. The transformation was, in some ways, the most important consequence of the InstructGPT paper. The paper did not just introduce a technique — it introduced a practice, and the practice became, within a year, the standard for the industry.

The legacy is also, finally, a reminder of the gap between research and product. InstructGPT was, by most accounts, a research paper — it was published on arXiv, it was written by researchers, and it was addressed to the research community. But the paper’s most consequential impact was not on the research community — it was on the product community. The paper demonstrated that a specific technique (RLHF) could transform a specific model (GPT-3) into a useful product (InstructGPT, and later ChatGPT). The demonstration was, in some ways, the most important thing about the paper — not the specific results, but the demonstration that alignment was not just a research concern but a product concern, and that the techniques of alignment could be used to build products that people would use. The lesson — that research papers can, sometimes, change the trajectory of an industry — is one that the AI field has taken to heart, and the InstructGPT paper is, by most accounts, one of the clearest examples of a research paper that did.

---

---

---

Was instructgpt inevitable — the product of forces too large to redirect — or was it a series of choices, each of which could have gone differently? The answer matters, because it determines whether the future is something that happens to us or something we make.