The GPT-2 ‘Too Dangerous to Release’ Decision, February 2019

---

The Model

GPT-2 was a significant step up from GPT-1. It had 1.5 billion parameters, compared to GPT-1’s 117 million — a thirteen-fold increase. It was trained on a much larger dataset: about 40 gigabytes of text from WebText, a dataset of web pages that OpenAI had collected from outbound Reddit links with a minimum karma score of 3 (a proxy for content quality, on the theory that Reddit users’ upvotes filter out the worst material). The model used the same architecture as GPT-1 — a Transformer decoder — but at a much larger scale.

The results were striking. GPT-2 could generate coherent paragraphs of text on virtually any topic. Given a prompt about unicorns, it would produce a plausible-sounding article about unicorns. Given a prompt about a scientific topic, it would produce a plausible-sounding explanation. The text was not always accurate — the model frequently made factual errors — but it was fluent, grammatically correct, and often indistinguishable from human-written text at first glance.

This was a significant capability. Previous language models had been able to generate text, but the text had been obviously machine-generated — full of grammatical errors, non-sequiturs, and incoherence. GPT-2 was the first model that could generate text that was, at least in short passages, indistinguishable from human writing.

---

The Decision

OpenAI’s decision to withhold the full model was unusual. In the research community, the standard practice was to publish models and code openly, so that other researchers could verify, build on, and extend the work. OpenAI broke with this practice, citing concerns about misuse.

The blog post argued that GPT-2 could be used to generate “deceptive, biased, or abusive language at scale.” Specifically, the post raised four misuse concerns: generating convincing fake news at scale, impersonating others online (generating text that appeared to come from a specific person), automating abuse or harassment (mass-producing abusive content), and automating spam or phishing messages. The post argued that the risks of releasing the full model outweighed the benefits, and that OpenAI had a responsibility to release the model in a “staged release” — starting with the smaller version and gradually releasing larger versions as they assessed the risks.

---

The Criticism

The decision was controversial. Critics argued that OpenAI was overstating the risks for commercial reasons — generating publicity, establishing OpenAI as a serious player in AI safety, and creating a narrative that would justify a closed-source approach for future models. The term “AI safety theatre” — a play on Bruce Schneier’s older phrase “security theatre” — was used to describe the decision: the suggestion that OpenAI was performing concern about safety without actually achieving meaningful safety outcomes.

Critics also argued that the decision was counterproductive. By withholding the model, OpenAI prevented the research community from studying the model’s capabilities and limitations, from developing defences against potential misuse, and from building on the work. Several independent researchers, using publicly available information about GPT-2’s architecture, built their own versions — most notably OpenGPT-2, a replication of the full 1.5B model built by two Brown University graduate students, Aaron Gokaslan and Vanya Cohen, on the OpenWebText corpus for approximately $50,000 of compute. The replication demonstrated that the model could not be kept secret: once the architecture was known, anyone with sufficient compute could rebuild it.

The decision also set a precedent. After GPT-2, OpenAI would adopt a closed-source approach for its most powerful models — GPT-3 was released in June 2020 only through an API, not as open weights (model parameters released publicly, allowing anyone to download and use the model). GPT-2’s full model was eventually released as open weights; GPT-3 was not. The GPT-2 “too dangerous to release” decision was the first step in the transformation of OpenAI from an open research lab to a closed commercial company — a transformation that would be completed over the following four years.

---

What It Meant

The GPT-2 decision was, in retrospect, a turning point. It was the moment when AI companies began to treat their models not as research artifacts to be shared, but as commercial assets to be controlled. The “too dangerous to release” framing would be used again — by OpenAI for GPT-3, by Google for some of its models, and by Anthropic (the AI company founded in 2021 by former OpenAI researchers, led by siblings Dario Amodei and Daniela Amodei) for some of its Claude models. The framing provided a safety justification for what was also, transparently, a commercial strategy.

The decision also revealed a tension that would define the AI industry: the tension between openness and control. OpenAI had been founded in 2015 as an open research lab, dedicated to sharing its work with the world. GPT-2 was the moment when that commitment began to erode. By the time ChatGPT was released on November 30, 2022, OpenAI had become one of the most secretive companies in the technology industry — and the “too dangerous to release” framing of GPT-2 was the first step in that transformation.

The feared harms of GPT-2 never materialised. The model was eventually released in full, and the world did not end. But the precedent was set, and the AI industry has been living with the consequences ever since.

---

---

What would change if more people understood the story behind the gpt-2 release decision? Who benefits from the current state of affairs, and who is left out? The conversation is worth having — with colleagues, with students, with anyone who uses technology without thinking about where it comes from.