isHistory
SeriesMinds & Machines📰 ArticleAct IV
A17Act IV · The Winter Thaws

The Race to AGI: When Silicon Valley Decided to Change the World

Ishaan25 min read
On this page32 sections

“This is the most important technology in the history of civilisation. It will be more significant than electricity, more significant than the internet. Everything changes. Everything. The people who build it are going to be the most powerful people in the world. And the question is whether those people are trying to make it good or not. I’m betting on people who are trying to make it good.”

— A venture capitalist, San Francisco, 2019

San Francisco, California. 2019. A venture capitalist is explaining to a journalist why he has put fifty million dollars into an AI startup with no product, no revenue, and a founding team of four researchers who had left a major technology company six months ago. The journalist is trying to understand the logic.

“The way I think about it,” the investor says, “is that this is the most important technology in the history of civilisation. It will be more significant than electricity, more significant than the internet. Everything changes. Everything.”

He pauses, in the manner of a person who has thought about this a great deal and is not finished.

“The people who build it are going to be the most powerful people in the world. And the question is whether those people are trying to make it good or not. I’m betting on people who are trying to make it good.”

The journalist asks whether he believes the startup will actually build AGI.

Important

“I don’t know,” the investor says, with what appears to be complete honesty. “But I believe they might. And if there’s a chance they might — if there’s even a real chance — you have to be in the room.”

This is the logic of the AI race: a combination of genuine belief in transformative potential, competitive anxiety about who will lead, and a specific theory of how to ensure the transformation is beneficial. The logic is not irrational. It has also produced competitive dynamics that the people who generated them are not fully in control of.

The Infrastructure of the Race: Capital, Compute, and Talent

The race to AGI is, at its material foundation, a competition for three resources: capital, computing, and talent.

Info

1. Capital

The development of frontier AI systems is extraordinarily expensive. Training a large language model at the scale of GPT-4 or its successors requires hundreds of millions to billions of dollars in computing costs alone, before accounting for the researchers who design and train the models, the engineers who build and maintain the infrastructure, and the operations required to deploy the resulting systems at scale.

The capital comes from several sources: venture capital (Sequoia, Andreessen Horowitz, Khosla), technology company balance sheets (Google, Microsoft, Meta, Amazon), sovereign wealth funds, and increasingly AI companies’ own revenue.

2. Computing

The hardware required for training and deploying large AI models — primarily NVIDIA’s GPU and TPU accelerators — has been in such high demand since the deep learning revolution that supply constraints have been one of the primary bottlenecks for AI development.

NVIDIA’s dominance of the AI hardware market gives the company extraordinary leverage over the AI ecosystem. The pricing of NVIDIA’s hardware, the allocation of its limited production capacity, and the roadmap for future hardware generations all affect the competitive dynamics of the AI industry.

3. Talent

The researchers who can design, train, and improve frontier AI systems are scarce — the number of people who have both the technical skills and the relevant experience to work at the absolute frontier of AI research is measured in the hundreds or low thousands worldwide. Competition for this talent has driven AI researcher compensation to levels that most academic institutions cannot match.

Important

The concentration of capital in a small number of frontier AI organisations is both a consequence of the capital requirements and a driver of them. Because training the largest models requires the most capital, the organisations with access to the most capital have a structural advantage in building the most capable models. Because the most capable models attract the most commercial applications, the organisations with the most capable models generate the most revenue.

The economic logic creates a flywheel: capital enables capability, capability generates revenue, revenue enables more capital.

The Organisations: Who Is in the Race

The AI race is not a race between every organisation working on AI — it is a race between a small number of organisations working at the absolute frontier of capability.

Info

Google / DeepMind

Google has been at the frontier of AI research since it acquired DeepMind in 2014 and established Google Brain in 2011. The combination of Google Brain and DeepMind — unified into Google DeepMind in 2023 — is the single organisation with the deepest AI research talent, the most computational resources, and the longest track record of frontier results. AlphaGo, AlphaFold, the Transformer architecture, BERT — the list of foundational contributions from Google’s AI organisations is extraordinary.

OpenAI

OpenAI’s specific contribution to the race dynamics was the deployment of ChatGPT — the product that made conversational AI mainstream and that triggered the competitive responses that define the current period.

Anthropic

Founded in 2021 by former OpenAI researchers, Anthropic has positioned itself as the safety-focused alternative — an organisation that prioritises research into AI alignment and interpretability alongside capability development. Anthropic’s Claude models have been competitive with OpenAI’s GPT series on many benchmarks and have developed a reputation for being more careful than GPT models in specific domains.

Meta AI

Meta’s AI research, led by Yann LeCun, has been distinguished from other frontier AI organisations by its commitment to open-source models. The LLaMA series — large language models whose weights have been publicly released — has made frontier-quality AI capabilities available to researchers and developers who cannot access the proprietary APIs of OpenAI or Google.

Mistral and other European players

Mistral AI, founded in Paris in 2023 by former Google DeepMind and Meta researchers, has established itself as the most capable European frontier AI organisation, with models that are competitive with much larger American models on efficiency-adjusted benchmarks.

Chinese organisations

Baidu, Alibaba, Tencent, and a large number of Chinese AI startups are competing at the frontier of AI capability in the Chinese market and, increasingly, internationally.

The Effective Altruism Connection and the AGI Framing

One of the most distinctive features of the AI race, compared to previous technology competitions, is the specific ideological context in which many of its participants operate — a context shaped by the effective altruism (EA) movement and by the specific set of concerns about existential risk from advanced AI that EA has promoted.

Definition

Effective altruism (EA) — A philosophical and practical movement that asks how individuals can do the most good with their resources — time, money, skills, career choices. In the early 2010s, the EA movement’s analysis led many of its participants to conclude that reducing existential risks — risks of catastrophic outcomes for humanity, including risks from advanced AI — was among the most important things individuals could work on.

Info

The EA influence on the AI race is visible in several ways:

  • The founding of OpenAI itself was partly motivated by concerns about the concentration of AI development in organisations without explicit safety commitments.
  • Several of OpenAI’s early employees and board members were affiliated with EA.
  • Anthropic’s founding was partly motivated by safety concerns that had roots in EA-influenced thinking.

The specific AGI framing — the view that AI systems are approaching a level of general capability that will be transformative and potentially existential — is also associated with EA.

Important

This framing has specific effects on how the race is conducted:

  • It creates a specific urgency — if AGI is coming soon, the decisions made now about who builds it and how are extraordinarily consequential.
  • It creates a specific theory of impact — being at the frontier and having influence over how AGI is built is more valuable than not being at the frontier and having less influence.
  • It creates a specific tension — the urgency to be at the frontier is in tension with the caution that the potential risks of AGI would seem to require.

The Safety-Capability Tradeoff: Real or Imagined?

One of the most persistently debated questions in the AI industry is whether safety and capability are fundamentally in tension — whether working on making AI systems safer necessarily comes at the cost of making them less capable, or whether safety and capability are more complementary than they appear.

Example

The pessimistic view: Safety research is inherently slower and more difficult than capability research, that organisations that prioritise safety will fall behind organisations that prioritise capability, and that the competitive dynamics of the AI race create pressure to prioritise capability over safety.

The optimistic view: Safety and capability are not fundamentally in tension — that the same research that makes AI systems more capable also makes them more reliable, more interpretable, and more controllable. The development of RLHF, for example, was simultaneously a capability improvement (models trained with RLHF followed instructions better) and a safety improvement (models trained with RLHF were less likely to produce harmful outputs).

Note

The more important point may be that the question is somewhat artificial. AI development organisations do not face a simple binary choice between safety and capability. They face a range of specific decisions about what to work on, how to train models, what to deploy, and at what pace. The aggregate effect of those decisions on both safety and capability depends on the specific choices made, not on an abstract tradeoff.

The Competitive Dynamics: How the Race Accelerates Itself

The AI race has specific competitive dynamics that create pressure toward faster development and more aggressive deployment — dynamics that are not always in the interest of the participants but that are hard to resist once they are established.

Important

1. The talent competition

Competition for AI research talent creates pressure to offer the most attractive compensation packages, the most exciting research problems, the most compute resources, and the most visible positions in the field. Organisations that fall behind in talent recruitment face a compounding disadvantage — without the best researchers, they cannot produce the best results, which makes them less attractive to the best researchers. Winner-takes-most dynamic.

2. The product competition

Once ChatGPT demonstrated that conversational AI was commercially viable, every organisation with AI capabilities faced pressure to deploy products before its competitors. The product competition creates pressure to deploy faster, to ship before all the safety questions are answered, to release capabilities that are impressive but imperfectly understood. The November 2022 ChatGPT launch was followed within months by Bard from Google, Copilot from Microsoft, and a flood of AI products from startups.

3. The funding competition

The AI race requires capital, and capital follows demonstrated results. Organisations that produce impressive results attract more capital, which enables more compute, which enables more impressive results. The funding competition creates pressure toward the kind of dramatic, publicisable demonstrations — record benchmark scores, impressive capability demonstrations, surprising emergent behaviours — that attract capital, rather than the slower, less publicisable work of building reliable, safe, well-understood systems.

4. The narrative competition

AI organisations compete not just for talent, capital, and customers but for the narrative frame through which their work is understood. The narrative frames available — from “democratising AI for everyone” to “building the most powerful technology in human history” to “solving the alignment problem before it’s too late” — all have commercial and political implications. The narrative competition creates incentives toward the most compelling stories, not necessarily the most accurate ones.

Warning

What is distinctive about the AI race is the combination of these competitive dynamics with the specific AI safety concerns that many participants acknowledge: the possibility that AI systems could cause serious harm if developed without adequate safety precautions, and the possibility that the race itself is accelerating the development of systems whose safety properties are not fully understood.

Demis Hassabis and the Scientific Vision

Demis Hassabis
Born:
July 27, 1976, London, England
Nationality:
British (Greek-Cypriot descent)
Role:
AI researcher, entrepreneur
Known for:
Co-founding DeepMind (2010); AlphaGo, AlphaFold, AlphaTensor, GNoME; Nobel Prize in Chemistry (2024) for AlphaFold; distinctive vision of AI as a tool for accelerating scientific discovery

One of the most interesting perspectives on the AI race comes from DeepMind’s co-founder and CEO, Demis Hassabis — a person whose vision of AI is distinctively oriented toward scientific discovery rather than toward commercial applications or the AGI race framing.

Important

Hassabis founded DeepMind with a specific vision: to use AI as a tool for accelerating scientific discovery. Not just to build AI systems that could perform tasks previously performed by humans, but to build AI systems that could discover scientific knowledge that humans had not yet found — to use AI to solve problems in biology, medicine, physics, and other sciences that human researchers, working at human speed with human methods, could not solve.

This vision produced:

  • AlphaFold — the protein structure prediction system that solved one of biology’s grand challenges
  • AlphaTensor — a system that discovered new matrix multiplication algorithms more efficient than those humans had developed
  • GNoME — a system that discovered millions of new stable materials
Note

Hassabis’s specific intellectual background — as a former chess prodigy, a game designer, and a computational neuroscientist — has shaped DeepMind’s research culture in specific ways. The organisation values intellectual breadth, scientific rigour, and the willingness to work on problems that require years of sustained effort without immediate commercial payoff.

The China Factor: A Parallel Race

The AI race is not exclusively American. China is engaged in a parallel and increasingly competitive AI development effort that has its own dynamics, its own leading organisations, and its own governmental context.

China’s “New Generation AI Development Plan”
Date:
July 2017
Location:
State Council of the People’s Republic of China
Significance:
China formally announced its AI ambitions — called for China to be the world’s leading AI nation by 2030
Outcome:
Substantial government funding for AI research and development; range of policies designed to build China’s AI ecosystem; established a parallel frontier AI development effort alongside the American one
Info

The Chinese AI ecosystem is different from the American one in several important respects:

  • Chinese technology companies — Baidu, Alibaba, Tencent, Huawei, and increasingly a large number of AI-focused startups — have access to the large domestic user bases and the vast quantities of data that large-scale AI training requires.
  • The Chinese government’s data governance regime, which has been more permissive than European approaches and more centrally directed than the American approach, has made data access relatively less constrained for Chinese AI development.
  • Chinese AI research has produced results that are competitive with the American frontier in many domains.
US export controls on advanced AI chips
Date:
October 2022 (initial), expanded subsequently
Location:
US Department of Commerce, Bureau of Industry and Security
Significance:
Export controls on NVIDIA’s most advanced AI chips designed to slow the development of Chinese AI capabilities by restricting access to the computing hardware that large-scale AI training requires
Outcome:
Most direct use of government power to shape the competitive dynamics of the AI race; whether effective or primarily accelerating Chinese investment in domestic semiconductor manufacturing is contested

The Small Teams That Matter: Startups and the Ecosystem

Alongside the major AI organisations, a large ecosystem of AI startups has developed that is producing commercially important applications and, in some cases, research that is influencing the direction of the field.

Info

Three types of AI startups:

  • Application companies — using large language model APIs from OpenAI or Anthropic to build specific applications in specific domains (healthcare AI, legal AI, educational AI). These companies are commercially important and, in aggregate, represent a large and growing market.

  • Infrastructure companies — building the tools, platforms, and services that make AI development accessible to organisations that don’t have the resources to build from scratch (vector databases, fine-tuning services, evaluation platforms).

  • Foundation model companies — attempting to develop models competitive with those from the major AI organisations. These companies — Mistral in Europe, Cohere in Canada, Inflection (before its effective dissolution), and others — are the most direct competitors to the major AI organisations and are developing some of the most technically interesting research in the field.

The Regulation Question: Government Enters the Race

The speed and the scale of the AI race have attracted significant governmental attention, and the regulatory landscape for AI has been changing rapidly in ways that will shape the competitive dynamics of the industry.

EU AI Act passed
Date:
2024
Location:
European Union
Significance:
The most comprehensive regulatory framework for AI yet enacted — classifies AI systems by risk level from “unacceptable risk” (prohibited) to “high risk” (requires conformity assessments) to lower-risk systems
Outcome:
Most capable general-purpose AI systems — frontier models developed by OpenAI, Google, Anthropic, and competitors — are subject to specific obligations under the AI Act
Biden executive order on AI
Date:
October 2023
Location:
White House, United States
Significance:
Required frontier AI developers to share safety test results with the government, established standards for AI security and safety, directed federal agencies to develop AI governance practices
Outcome:
Ambitious in scope but limited in enforceability — an executive action rather than legislation
Bletchley AI Safety Summit
Date:
November 2023
Location:
Bletchley Park, UK
Significance:
First international AI Safety Summit — venue chosen deliberately for its association with the wartime codebreaking that has been incorporated into AI’s origin story
Outcome:
Produced the Bletchley Declaration, signed by twenty-eight countries, acknowledging the potential risks of frontier AI systems and committing the signatories to cooperation on AI safety

What Is Actually Being Built: The Gap Between Hype and Reality

The AI race has been accompanied by an extraordinary amount of hype — predictions about imminent AGI, claims about AI capabilities that exceed what the systems can actually do, and a general atmosphere of excitement that can obscure the genuine uncertainties and the genuine limitations of current systems.

Example

What current systems can do

Large language models can generate fluent, contextually appropriate text on essentially any topic. They can write code, explain concepts, summarise documents, translate languages, answer questions, and engage in extended conversations. They can perform well on many standardised tests and benchmarks. They can, in some domains and for some tasks, produce outputs that are comparable to those of skilled human practitioners.

What current systems cannot do reliably

Current large language models cannot reliably reason about multi-step problems that require careful tracking of intermediate states. They cannot reliably distinguish what they know from what they do not know — they are prone to hallucination, producing confident-sounding outputs that are factually incorrect. They cannot reliably generalise from training distribution to novel situations. They cannot reliably take sequences of actions in the world to achieve complex goals.

What the near-term trajectory is likely to produce

The scaling laws suggest that continued investment in larger models trained on more data will produce further capability improvements, though the specific nature of those improvements is hard to predict in advance. The research directions being actively pursued — better reasoning, improved factual reliability, more robust generalisation, more capable agentic behaviour — are likely to produce meaningful progress.

What remains genuinely speculative

Whether current AI architectures can produce the kind of general intelligence that would qualify as AGI by any reasonable definition is genuinely contested. The specific capabilities that would distinguish AGI from very capable but domain-limited AI — genuine causal reasoning, robust common-sense understanding, flexible generalisation to truly novel situations, genuine understanding rather than sophisticated pattern matching — are not clearly present in current systems and it is not clear whether they will emerge from scaling current approaches or will require architectural innovations.

The Talent That Left and Where It Went

One of the most interesting and most revealing dynamics of the AI race is the pattern of talent movement — the specific people who left major AI organisations and what they built when they left.

Info

Ilya Sutskever, co-founder of OpenAI and its Chief Scientist from the beginning until 2024, left OpenAI in May 2024 to found Safe Superintelligence Inc. (SSI) — an organisation whose name itself is a mission statement, and whose specific goal was to develop the safety properties required for superintelligent AI before that AI was built.

The researchers who left OpenAI to found Anthropic — Dario Amodei, Daniela Amodei, Chris Olah, Tom Brown, and others — brought with them not just technical expertise but specific concerns about whether OpenAI was moving too fast and investing adequately in safety. Anthropic’s founding was explicitly motivated by these concerns.

The founders of Mistral AI — Arthur Mensch, Guillaume Lample, and Timothée Lacroix — left DeepMind and Meta to build a European frontier AI organisation. Their specific contribution has been demonstrating that smaller, more efficient models can compete with much larger models on many benchmarks.

Note

The pattern of talent movement reveals something about the diversity of perspectives within the AI research community — and about the genuine disagreements about what to build, how to build it, and at what pace. The organisations that former OpenAI researchers have founded represent a range of positions from “we need to build AGI as safely as possible and as fast as possible” (Sam Altman’s OpenAI) to “we need to build AGI safely and with more caution than current dynamics allow” (Sutskever’s SSI, Amodei’s Anthropic).

The Investors: Who Is Funding the Race and Why

The AI race has attracted capital from an extraordinary range of investors, motivated by a combination of financial opportunity and genuine belief in the technology’s transformative potential.

Important

What is most interesting is the specific willingness to invest in organisations that explicitly discuss the risks of the technology they are building — the investors who are putting money into Anthropic or OpenAI despite (or because of) the organisations’ explicit acknowledgment that they may be building technology that could be catastrophic if developed incorrectly.

The investor reasoning in these cases is a specific form of Pascal’s Wager: if AI is going to be developed anyway, and if the development of very powerful AI carries real risks, then it is better to have safety-focused organisations at the frontier than to leave the frontier to organisations that are less focused on safety. The investment is motivated not by the desire to profit from catastrophe but by the belief that safety-focused organisations at the frontier are better for humanity than the alternative.

Warning

Whether this reasoning is correct — whether the presence of safety-focused organisations at the frontier actually reduces the risks of AI development or whether it primarily accelerates the development of systems whose safety is not fully understood — is the central contested question about the AI race.

The Deepest Question: Is the Race Necessary?

Underlying all the specific dynamics of the AI race is a fundamental question: is the race necessary?

Example

The argument that it is necessary: AI development is proceeding rapidly because the technology is advancing, the commercial incentives are powerful, and many talented people are working on it. Given that this development is happening, it is better to have safety-focused organisations at the frontier — organisations that are investing in alignment research, that are committed to responsible deployment, that are engaging with regulatory questions in good faith — than to leave the frontier to organisations with less focus on safety.

The argument that it is not necessary: The AI race itself is one of the primary drivers of the rapid development that creates risks. If the major AI organisations committed to slower, more careful development — if they collectively agreed to proceed at a pace that allowed safety research to keep up with capability research — the risks would be lower. The competitive dynamics that individual organisations cannot resist when acting alone could be managed through collective agreements. The race is necessary only because the organisations competing in it have chosen to compete, and those choices are not dictated by any external force.

Important

The honest answer is that both arguments have merit, and the correct response is somewhere between them. Some competition is productive — it drives the innovation that makes AI more useful and, potentially, safer. Some competition is destructive — it creates pressure to deploy systems before their safety is fully understood, to prioritise speed over caution, to treat safety research as a constraint to be managed rather than a core objective.

The challenge is to preserve the productive competition while limiting the destructive competition.

This is a governance challenge as much as a technical one. The regulatory frameworks that are being developed, the voluntary commitments that frontier AI organisations are making, the international cooperation that is being pursued — all are attempts to manage the competitive dynamics in ways that preserve the benefits while limiting the harms.

Quote

Whether these governance efforts will be adequate — whether they will be sufficient to ensure that the development of very powerful AI goes well for humanity — is the most important question in the history of AI. And it is a question that the race itself is, in a specific sense, making harder to answer by the day.

The Legacy of the Race: Whatever Happens Next

The AI race of 2015-2025 will be remembered, whatever happens next, as one of the most consequential periods in the history of technology.

Info

  • The systems that have been built — the large language models, the protein structure prediction systems, the multimodal models, the autonomous agents — have already transformed how people work, communicate, learn, and think. This transformation is still in its early stages.

  • The organisations that have competed in the race — OpenAI, Google/DeepMind, Anthropic, Meta, and their competitors — have defined the frontier of what is possible in AI and have established the specific competitive dynamics that will shape AI development for the foreseeable future.

  • The people who have led the race — Sam Altman, Demis Hassabis, Dario and Daniela Amodei, Yann LeCun, Jeff Dean, and the dozens of researchers and executives who have shaped the organisations — will be among the most consequential people of their generation, for better or for worse.

Warning

What is genuinely uncertain — what history has not yet revealed — is whether the race will have been a good thing. The technology it has produced is powerful and useful and is changing the world. Whether it will change the world for the better — whether the benefits will exceed the harms, whether the safety research will prove adequate, whether the governance frameworks will be sufficient — depends on decisions that are still being made and on developments that have not yet occurred.

Quote

The race is not over. It is accelerating. The people running it know this, and most of them are trying, in their specific ways and with their specific theories, to ensure that when it reaches its destination, the destination is somewhere worth going.

Whether they are succeeding is the question that the next chapter of this story will answer.

Further Reading

Further Reading
  • “Power and Progress: Our Thousand-Year Struggle Over Technology and Prosperity” by Daron Acemoglu and Simon Johnson (2023) — A historical and economic analysis of how technology affects inequality and power, providing essential context for thinking about who benefits from the AI race.
  • “Human Compatible” by Stuart Russell (2019) — Russell’s account of the AI alignment problem and his proposed solutions, which have influenced how many race participants think about the safety challenges they face.
  • “The Age of AI: And Our Human Future” by Henry Kissinger, Eric Schmidt, and Daniel Huttenlocher (2021) — A geopolitical perspective on AI development, particularly relevant to understanding the US-China dynamics of the AI race.
  • “The Coming Wave” by Mustafa Suleyman and Michael Bhaskar (2023) — Suleyman, co-founder of DeepMind, provides an insider’s account of the AI race and its implications.
  • “Atlas of AI” by Kate Crawford (2021) — A critical examination of the material and political dimensions of AI development — the labour, the resources, the power structures — that the more optimistic accounts of the AI race tend to obscure.
Part 18: The Alignment Problem — Can We Build AI That Wants What We Want?

The full intellectual story of AI alignment research — from Norbert Wiener’s early warnings to the technical work of the current era, from the philosophical foundations to the specific technical approaches. The most important unsolved problem in AI, and the people trying to solve it.

Comments

Reply on Bluesky → (opens in a new tab)