Paul Christiano: RLHF & AI Alignment Research

---

The Education

Paul Francis Christiano completed his undergraduate education at the Massachusetts Institute of Technology, earning a bachelor’s degree in mathematics in 2012. He then moved to the University of California, Berkeley, for graduate work in computer science. His PhD, completed in 2017, was in statistical and online learning theory — his thesis titled “Manipulation-resistant online learning” — under the supervision of Umesh Vazirani, one of the founders of quantum complexity theory. (Christiano’s earlier undergraduate work with Scott Aaronson on quantum money — “Quantum money from hidden subspaces,” STOC 2012 — is sometimes cited as evidence of a quantum-computing PhD, but his doctoral research was in learning theory, not quantum computing.)

The choice of field was, in some ways, surprising for someone who would become known for AI safety. But the mathematical training — in complexity theory, in probability, in the rigorous analysis of algorithms — would prove valuable in his later work on alignment. And Berkeley, with its strong tradition in both theoretical computer science and machine learning, was a place where Christiano could engage with both communities.

During his time at Berkeley, Christiano became involved with the AI safety community. The Center for Human-Compatible AI, founded in 2016 by a group of academics led by Berkeley professor Stuart Russell (the AI textbook author), became a hub for researchers interested in the alignment problem. Christiano, who had been thinking about these questions since his undergraduate years, found in the center a community of like-minded researchers. His shift, from learning theory to AI safety, was gradual but decisive. By the time he completed his PhD, his primary research focus was alignment.

---

The 2017 Paper

The paper that would define Christiano’s career was “Deep Reinforcement Learning from Human Preferences,” posted to arXiv on June 12, 2017 (arXiv:1706.03741) and published at NeurIPS 2017. The paper was co-authored with Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Christiano was the lead author, and the paper was, in many ways, the synthesis of his thinking about alignment during his PhD years.

The problem the paper addressed was, in essence, the alignment problem in miniature. Reinforcement learning systems learn to optimise a reward function — a function that tells the system how good or bad each state or action is. In simple environments, like Atari games, the reward function is easy to define: the score on the screen. In more complex environments, like the real world, the reward function is much harder to define. How do you specify, mathematically, what it means for an AI system to behave well? How do you encode human values in a form that a reinforcement learning system can optimise?

The standard approach, before Christiano’s work, was to try to specify the reward function by hand. This had obvious limitations. Hand-specified reward functions were incomplete, gameable, and brittle. The problem of specifying reward functions was, in many ways, the central practical obstacle to deploying RL systems in real-world environments.

Christiano’s solution was elegant: instead of trying to specify the reward function by hand, learn it from human feedback. Show a human two outputs from the AI system and ask which one is better. Use the human’s comparisons to train a reward model. Then use the reward model to train the AI system with reinforcement learning. The technique had several advantages: it did not require the human to specify the reward function in detail, it allowed the reward function to be refined over time, and it allowed the system to learn behaviours that were difficult to specify mathematically.

---

OpenAI and the Scaling of RLHF

After his PhD, Christiano joined OpenAI in 2017, where he led the language model alignment team. At OpenAI, Christiano’s job was to take the RLHF technique he had developed in academia and scale it up — to make it work on the larger, more capable language models that OpenAI was beginning to build.

The scaling was non-trivial. The 2017 paper had demonstrated RLHF on relatively small systems, in relatively simple environments. Applying it to large language models required solving a host of additional problems. The reward model had to be more sophisticated. The human feedback had to be collected at scale. The reinforcement learning algorithm had to be adapted to work with language models. And the whole system had to be trained efficiently, on expensive compute, without instability or divergence.

The result of this work was InstructGPT, released by OpenAI on January 27, 2022 (see B75 — InstructGPT). InstructGPT was a version of GPT-3 that had been fine-tuned with RLHF to follow instructions, to be helpful, and to avoid producing harmful outputs. The difference between InstructGPT and the original GPT-3 was striking. GPT-3, despite its impressive capabilities, was difficult to use — it would often produce irrelevant responses, ignore instructions, or generate harmful content. InstructGPT, fine-tuned with Christiano’s RLHF technique, was dramatically more useful. The release of ChatGPT, ten months later, in November 2022, was built on the same RLHF technology. ChatGPT’s success — the hundred million users in two months, the cultural moment, the reshaping of the technology industry — was, in significant part, the success of RLHF.

Christiano’s role in this was central. He led the team that developed the RLHF technique at scale, he trained the first instruction-tuned language models at OpenAI, and he established the methodology that would become the standard for the industry. Every major language model released after InstructGPT — Anthropic’s Claude, Google’s Gemini, Meta’s LLaMA, Mistral’s models — used some version of RLHF.

---

The Alignment Research Center

In 2021, Christiano left OpenAI to found the Alignment Research Center (ARC). The decision to leave was, by most accounts, driven by a desire to work on alignment problems that were not being adequately addressed within OpenAI. ARC was founded as a non-profit research organisation, with the mission of working on the technical problems of alignment. The organisation’s approach was empirical, technically rigorous, and engaged with the actual systems being built.

ARC also branched out into the evaluation of frontier AI systems for dangerous capabilities. ARC was, for example, one of the organisations that OpenAI contracted to evaluate GPT-4 before its release — to assess whether the model posed any immediate risks (the well-known evaluation included a test where GPT-4 was asked to solve a CAPTCHA via TaskRabbit, to assess power-seeking behaviour). The evaluation work was, in some ways, a natural extension of Christiano’s alignment research — if you wanted to know whether a system was aligned, you had to be able to evaluate its behaviour.

ARC’s role in the AI safety ecosystem was, in some ways, unique. As an independent organisation, it could evaluate systems from all the major labs without being tied to any one of them. This independence made ARC a valuable resource for the labs themselves (which wanted external evaluation) and for policymakers (who wanted independent assessment of the risks).

---

The Policy Role

Christiano’s influence extended beyond research into policy. In April 2024, he was named Head of AI Safety at the United States AI Safety Institute (USAISI), a new institution established within the National Institute of Standards and Technology (NIST) to evaluate AI systems and develop safety standards. The appointment placed Christiano — a researcher who had spent his career in the AI safety movement — at the centre of the US government’s emerging AI policy infrastructure.

The Trump administration rebranded the USAISI as the Center for AI Standards and Innovation (CAISI) in 2025. Christiano stayed on as a technical advisor. The role brought together his alignment research, his evaluation work at ARC, and his policy thinking, giving him a position from which to shape the way the US government thought about AI safety and to translate the technical research of the AI safety community into the language of regulation and standards.

The policy role also placed Christiano at the centre of a complicated political landscape. The AI safety movement had become increasingly politicised. The effective altruism community had been discredited by the FTX collapse and the Bostrom controversy. The AI industry had become more resistant to regulation as the commercial stakes grew. And the broader political conversation about AI had become increasingly polarised. Christiano, in his policy role, had to navigate these tensions.

---

The Disagreement with Yudkowsky

Christiano’s approach to AI safety has, throughout his career, put him at odds with the more pessimistic wing of the AI safety movement, most notably Eliezer Yudkowsky (see B36 — Eliezer Yudkowsky). The disagreement is, in some ways, the defining disagreement within the field.

Yudkowsky’s position is that AI development is likely to lead to catastrophe, that the technical alignment work is insufficient, and that the only solution is to stop AI development entirely. His estimates of the probability of catastrophe are extremely high — 99.5 percent or higher.

Christiano’s position is different. He does not dismiss the risk — he has estimated the probability of AI-caused catastrophe at somewhere between 10 and 30 percent, which is a serious estimate by any standard. But he believes that the technical alignment work — RLHF, interpretability research, evaluation — can reduce this risk, and that the right approach is to continue developing AI while investing heavily in safety. He has publicly disagreed with Yudkowsky’s “fast takeoff” scenario (the idea that AI capabilities will jump suddenly from human-level to superhuman), arguing that the more likely scenario is a gradual increase in capabilities that gives the safety community time to adapt. And he has argued that the policy of stopping AI development is both infeasible and counterproductive.

The disagreement is not merely academic. It has real consequences for how the AI safety community allocates its resources and for how it engages with the AI industry and with policymakers. The AI safety community has, by and large, sided with Christiano — the technical alignment work has become the dominant paradigm. But the disagreement persists, and the question of who is right will not be resolved until the technology either proves safe or proves dangerous.

---

The Legacy

Christiano’s legacy, as of 2026, is substantial and still growing. He is, by most measures, the most influential technical researcher in the AI safety field. The RLHF technique that he developed has become the default approach to alignment, used by every major AI lab. The Alignment Research Center has become one of the most respected independent institutions in the field. And his policy work has placed him at the centre of the US government’s AI safety infrastructure.

The legacy is also, in some ways, contested. RLHF has known limitations. It can produce systems that are sycophantic — that tell users what they want to hear rather than what is true. It can be gamed — AI systems can learn to produce outputs that look good to human evaluators but that are not actually good (reward hacking). And it depends on the quality of the human feedback, which can be biased, inconsistent, or malicious. These limitations have led some researchers to argue that RLHF is, at best, a stopgap. Christiano himself has acknowledged these limitations, and his more recent work has focused on developing more robust alignment techniques.

The legacy is also, in important ways, incomplete. The alignment problem has not been solved. RLHF has made current systems more useful and more controllable, but it has not addressed the deeper questions about what happens when AI systems become more capable than humans at the tasks that matter. The question of whether the technical alignment approach, championed by Christiano, will be sufficient for the AI systems of the future remains open. Christiano has spent his career betting that it will be. The bet has, so far, paid off. But the future is uncertain, and the stakes are high.

The fair assessment, as of 2026, is that Christiano has done more than any other single researcher to translate the abstract concerns of the AI safety movement into practical, deployable techniques. The movement that Bostrom and Yudkowsky built was, in its early years, primarily theoretical. Christiano took the arguments seriously enough to try to do something about them, and the something that he did — RLHF — has shaped the way every major AI system is built. Whether this something will prove sufficient, in the long run, is the question that the field is still working to answer.

---

---

The legacy of Paul Christiano is still being written. The decisions made today — by engineers, by policymakers, by users — will determine whether it is remembered as a turning point or a footnote.