The Governance Gap: Can Humanity Govern What It Has Built?
On this page13 sections
- The Size of the Gap
- The EU AI Act: The Most Consequential Framework
- The American Approach: Voluntary and Executive
- The International Coordination Challenge
- The Democratic Deficit: Who Governs AI Governs Everything
- The Speed Problem: Governance Always Lags
- The Frontier Model Forum: Industry Self-Governance
- The Nuclear Analogy: How We Governed the Last Dangerous Technology
- The Governance Proposals: What the Research Community Suggests
- What Good Governance Would Look Like
- The Urgency: Why This Cannot Wait
- The Honest Assessment: How Much Time We Have
- Further Reading
“The systems that are supposed to help us are being built without us.”
Geneva, Switzerland. June 2025. A diplomat from a small island nation in the Pacific is presenting testimony to a working group at the United Nations. The island nation has a population of eleven thousand people and a land area of twenty-one square kilometres. It is, by most measures, among the least powerful nations on earth.
The diplomat is talking about AI.
- Date:
- June 2025
- Location:
- United Nations, Geneva, Switzerland
- Significance:
- A small Pacific island nation — among the least powerful on earth — presented testimony on how AI climate-modelling systems, trained primarily on data from high-income countries, perform worse on the climate patterns that affect their survival
- Outcome:
- A specific, concrete illustration of the broadest challenge in AI governance: the people who will be most affected by AI development are not the people who are shaping it
Specifically, she is talking about the use of AI in climate modelling and in the administration of climate adaptation funds — funds that her country depends on for its survival, because the sea level is rising and the storms are getting stronger and the land is running out. The AI systems that model climate outcomes, she says, were trained primarily on data from high-income countries. They perform worse on the specific climate patterns that affect her country. The administration of climate funds increasingly uses AI to assess applications, and the AI systems favour applications that match the patterns in their training data — applications from organisations in countries that look like the training data.
“The systems that are supposed to help us,” she says, “are being built without us.”
She is not wrong. And her testimony is a specific and concrete illustration of the broadest challenge in AI governance: the people who will be most affected by AI development are not the people who are shaping it. The governance gap is not just technical. It is a gap between who has power and who bears consequences.
She is not wrong. And her testimony is a specific and concrete illustration of the broadest challenge in AI governance: the people who will be most affected by AI development are not the people who are shaping it. The governance gap is not just technical. It is a gap between who has power and who bears consequences.
The Size of the Gap
The governance gap — the distance between the pace of AI capability development and the pace of governance development — is large and growing. Understanding its size requires understanding both what governance exists and what governance is needed.
What exists. The governance infrastructure for AI that exists as of 2025 includes:
- The EU AI Act — the most comprehensive binding legal framework for AI yet enacted, establishing risk-based requirements for AI systems deployed in the European Union, with specific obligations for high-risk systems and general-purpose AI.
- The US AI Safety Institute, established within NIST following the Biden executive order, which conducts evaluations of frontier AI systems and develops safety standards, without enforcement authority.
- The UK AI Safety Institute, established following the Bletchley Summit, with similar evaluation functions and similar limitations on enforcement.
- Equivalent safety institutes in Japan, Canada, Australia, Singapore, and several other countries, coordinating through informal networks.
- Voluntary commitments from major AI companies — Frontier Model Forum members, signatories to the White House voluntary commitments — that include safety evaluation before deployment and commitments not to support specific harmful applications.
- A patchwork of national and subnational AI regulations addressing specific applications — facial recognition, algorithmic decision-making in employment and lending, autonomous vehicles — without a coherent overall framework.
- International discussions at the G7, G20, OECD, and UN levels that have produced principles and guidelines without binding commitments.
What is needed. The governance infrastructure that would be adequate to the challenge of managing frontier AI development would need to address:
- Mandatory safety evaluation requirements — binding obligations on frontier AI developers to evaluate their systems for dangerous capabilities before deployment, with independent verification of those evaluations.
- International coordination mechanisms — frameworks that can manage the competitive dynamics between the United States, China, and other major AI powers in ways that prevent a race to the bottom on safety.
- Democratic accountability — mechanisms through which the people most affected by AI development can have meaningful influence over how it proceeds, not just the governments and companies that currently control it.
- Liability frameworks — clear legal frameworks establishing who is responsible for AI-caused harms and what remedies are available.
- Adaptive governance — regulatory frameworks that can keep pace with the speed of AI development, updating as capabilities change rather than lagging by years.
The gap between what exists and what is needed is the governance gap. It is not a gap that can be closed quickly, but it is a gap that is growing rather than shrinking.
The EU AI Act: The Most Consequential Framework
The European Union’s AI Act, which entered into force in August 2024 and is being implemented on a phased schedule through 2026, is the most significant AI governance achievement to date and deserves extended analysis.
- Date:
- August 1, 2024
- Location:
- European Union
- Significance:
- First comprehensive binding legal framework for AI; established the risk-tiered approach that has become the template for AI regulation worldwide
- Outcome:
- Phased implementation through 2026; extraterritorial reach (any company deploying AI in the EU must comply); fines of up to €35 million or 7% of global turnover for the most serious violations
The EU AI Act’s four risk tiers:
- Unacceptable risk (prohibited): AI used for social scoring by governments, AI that manipulates people’s behaviour unconsciously, real-time remote biometric identification in public spaces by law enforcement
- High risk (specific requirements before deployment): AI used in critical infrastructure, education and vocational training, employment and workers management, access to essential services, law enforcement, migration control, and the administration of justice — subject to conformity assessments, documentation requirements, data governance standards, human oversight mechanisms, and post-market monitoring
- Limited risk (transparency required): disclosure that users are interacting with AI
- Minimal risk (unregulated)
General-purpose AI models above a computational threshold (10^25 floating-point operations in training) are designated “frontier models” and subject to the most stringent requirements, including systemic risk assessments, adversarial testing, incident reporting, and cybersecurity standards.
The Act takes a risk-based approach, classifying AI systems into four risk tiers. Unacceptable risk systems — AI used for social scoring by governments, AI that manipulates people’s behaviour unconsciously, real-time remote biometric identification in public spaces by law enforcement — are prohibited. High-risk systems — AI used in critical infrastructure, education and vocational training, employment and workers management, access to essential services, law enforcement, migration control, and the administration of justice — are subject to specific requirements before deployment: conformity assessments, documentation requirements, data governance standards, human oversight mechanisms, and post-market monitoring. Limited-risk systems require transparency — disclosure that users are interacting with AI. Minimal-risk systems are unregulated.
General-purpose AI models — the large language models and other foundation models that power most AI applications — are subject to specific requirements reflecting their cross-cutting potential for impact. Models above a computational threshold (10^25 floating-point operations in training) are designated “frontier models” and subject to the most stringent requirements, including systemic risk assessments, adversarial testing, incident reporting, and cybersecurity standards.
What the AI Act achieves. The AI Act achieves several important things that no previous AI governance framework had achieved.
It establishes binding legal requirements rather than voluntary commitments. Companies that deploy AI systems in the EU must comply with the Act’s requirements or face fines of up to €35 million or 7% of global turnover for the most serious violations.
It applies extraterritorially — not just to EU companies but to any company that deploys AI systems in the EU market. This gives the Act significant global reach, because most major AI companies serve European customers and must comply with the Act regardless of their own national jurisdiction.
It establishes the principle that AI systems must be evaluated for safety before deployment in high-stakes applications, not after harms occur. The conformity assessment requirements mean that the burden of demonstrating safety falls on developers and deployers, not on victims of AI failures.
It creates institutional infrastructure for AI governance — enforcement agencies in each member state, a European AI Office at the EU level, and the AI Board coordinating between member states — that provides ongoing oversight rather than one-time approval.
What the AI Act does not achieve. The AI Act’s limitations are also significant.
It does not establish the kind of international coordination that the competitive dynamics of AI development require. The Act creates requirements for AI deployed in the EU, but it cannot prevent the development of AI systems outside the EU or the deployment of those systems in jurisdictions without equivalent requirements.
The enforcement infrastructure is new and untested. The national market surveillance authorities responsible for enforcing the Act are in the early stages of building the technical capacity required to evaluate frontier AI systems. The gap between the legal requirements and the enforcement capacity may be large, particularly for the most technically complex frontier systems.
The definition of “general purpose AI” and the specific requirements for frontier models have been contested, and the lobbying of major AI companies during the Act’s development has shaped its provisions in ways that may limit its effectiveness.
The Act’s focus on deployed systems means it has limited reach over the research and development phase of AI development — the phase during which the most consequential decisions about system design and training are made.
The American Approach: Voluntary and Executive
The United States has not enacted comprehensive AI legislation as of 2025. The American approach to AI governance has relied primarily on executive action, voluntary commitments from AI companies, and sector-specific regulation by existing agencies.
- Date:
- October 30, 2023
- Location:
- Washington, D.C., USA
- Significance:
- The most comprehensive federal AI governance action in the United States to date — requiring frontier AI developers to share safety test results with the government and directing agencies to develop AI governance standards
- Outcome:
- Provisions were modified and selectively enforced by the Trump administration that took office in January 2025; the order’s enforcement mechanisms were limited by their reliance on voluntary reporting from AI companies
- Date:
- July 21, 2023
- Location:
- White House, Washington, D.C., USA
- Significance:
- Seven leading AI companies (Amazon, Anthropic, Google, Inflection, Meta, Microsoft, OpenAI) made voluntary commitments on AI safety, security, and trust — including safety evaluation before deployment and content authentication
- Outcome:
- Voluntary commitments depend on the good faith of the signatories and cannot be enforced against companies that violate them — subject to the same limitations as all voluntary governance
The Biden administration’s October 2023 Executive Order on AI was the most comprehensive federal AI governance action to date. The order required frontier AI developers to share safety test results with the government — specifically, results on systems that could pose risks to national security, economic security, or public health and safety. It directed agencies to develop AI governance standards for their specific domains. It initiated a process for developing AI technical standards through NIST. And it directed the development of international AI governance norms through diplomacy.
The order’s limitations were significant. Executive orders can be reversed by subsequent administrations, and the specific provisions of the Biden AI order were modified and selectively enforced by the Trump administration that took office in January 2025. The order’s enforcement mechanisms were limited — it required voluntary reporting from AI companies rather than mandatory disclosure. And the order’s scope was limited by the Constitutional and administrative law constraints that govern executive action.
The voluntary commitments that the Biden White House secured from major AI companies in July 2023 — covering safety evaluation, content authentication, and specific harmful application prohibitions — represented a parallel track to the executive order. The voluntary nature of these commitments meant they were subject to the same limitations as all voluntary governance: they depend on the good faith of the signatories and cannot be enforced against companies that violate them.
The sector-specific regulatory approach has produced a patchwork of AI-relevant requirements across different domains. The FDA has developed guidance on AI in medical devices. The CFPB has issued guidance on AI in credit decisions. The EEOC has issued guidance on AI in employment. These sector-specific frameworks address AI in specific contexts but do not provide a coherent overall governance framework.
The American approach reflects specific features of the political economy of AI governance in the US context: the significant lobbying power of the major AI companies, the constitutional constraints on regulatory authority, the institutional fragmentation of the regulatory system, and the specific political dynamics of a highly polarised political environment.
The International Coordination Challenge
The governance challenge that is hardest and most consequential is the international coordination challenge: how to manage AI development in the context of great-power competition between the United States and China, and how to extend effective governance to the global diversity of countries and communities that AI will affect.
The US-China dynamic. The competitive relationship between the United States and China in AI development is the central geopolitical fact of AI governance. Both countries have significant AI capabilities, both are investing heavily in AI development, and both have significant national security interests in AI. The competitive dynamic creates specific pressures against the kind of coordination that effective global AI governance would require.
The specific mechanisms of US-China AI competition — export controls on AI hardware, restrictions on AI research collaboration, the bifurcation of the global AI ecosystem into American and Chinese technology stacks — create structural obstacles to coordinated AI governance. Countries that use Chinese AI infrastructure are not subject to American governance requirements; countries that use American AI infrastructure are not subject to Chinese governance requirements. The bifurcation of the global AI ecosystem means that global governance frameworks must somehow bridge two technology ecosystems that are actively competing.
- Date:
- November 2, 2023
- Location:
- Bletchley Park, Buckinghamshire, United Kingdom (AI Safety Summit)
- Significance:
- 28 countries (including the United States and China) signed a declaration acknowledging the risks of frontier AI and committing to international cooperation on AI safety — a rare moment of US-China cooperation on AI
- Outcome:
- The specific content of cooperation has been limited, but the fact of the cooperation — that the two most competitive AI powers could agree on a common statement about safety — was itself significant
The Bletchley Declaration of November 2023 — signed by both the United States and China — represented a rare moment of US-China cooperation on AI. The Declaration acknowledged the risks of frontier AI systems and committed the signatories to cooperation on AI safety. The specific content of the cooperation has been limited, but the fact of the cooperation — that the two most competitive AI powers could agree on a common statement about safety — was itself significant.
The global south. The countries of the global south — Africa, Latin America, South Asia, Southeast Asia — face a specific governance challenge: they are not major developers of frontier AI, but they will be profoundly affected by AI systems developed primarily in the United States and China. The governance frameworks being developed in the United States and Europe reflect the specific concerns and the specific institutions of those regions, not necessarily the concerns and institutions of the global south.
The diplomat from the Pacific island nation was articulating a real problem: the AI systems that will shape global climate governance, global economic systems, global public health infrastructure — these systems are being developed by a small number of organisations in a small number of countries, and the governance frameworks that constrain them are primarily national frameworks in those same countries.
Building AI governance frameworks that include the global south is a challenge that the existing governance conversations have addressed inadequately. The United Nations-based processes — the UN Secretary General’s AI Advisory Body, the Multistakeholder Forum on Internet Governance — represent attempts to create inclusive governance conversations, but they have produced principles without enforcement mechanisms.
The Democratic Deficit: Who Governs AI Governs Everything
One of the most important critiques of current AI governance is the democratic deficit — the observation that the decisions being made about how AI is developed and deployed are not being made through processes that are democratically accountable to the people who will be most affected.
The major AI companies — OpenAI, Google, Anthropic, Meta, Microsoft — are private corporations primarily accountable to their shareholders and investors, with secondary accountability to the governments of their home jurisdictions. The people who will be most affected by AI systems — the workers whose jobs may be displaced, the communities subjected to AI surveillance, the patients whose medical decisions are informed by AI systems, the citizens whose political information environment is shaped by AI-generated content — have limited formal influence over how AI is developed and deployed.
The existing regulatory frameworks partially address this deficit. The EU AI Act’s requirements for transparency and human oversight create some mechanisms for accountability. The sector-specific regulatory frameworks in the United States create some accountability to affected communities through the regulatory process. The voluntary commitments of AI companies include some community engagement provisions.
But the democratic accountability of AI governance remains limited relative to the significance of the decisions being made. The specific decisions about which AI capabilities to develop, which systems to deploy, what safety standards to apply, and what beneficial applications to prioritise — these decisions are being made primarily by the engineers, researchers, executives, and investors of a small number of organisations, with limited democratic oversight.
This democratic deficit is not inevitable. Governance frameworks can be designed to include more democratic participation — through requirements for community consultation in AI deployment decisions, through democratic oversight of AI-using public institutions, through regulatory frameworks that give affected communities formal standing in regulatory processes. The design of such frameworks requires political will and institutional creativity that the current governance conversation has not yet fully mustered.
The Speed Problem: Governance Always Lags
One of the structural challenges of AI governance is the speed problem: governance processes — legislative, regulatory, judicial — are slower than technological development, and the governance that addresses the technology of today may be inadequate for the technology of tomorrow.
The EU AI Act took approximately three years from the European Commission’s initial proposal to its final adoption. In those three years, the capabilities of AI systems changed dramatically — the initial proposal was written before ChatGPT existed, and significant revisions were required to address the general-purpose AI capabilities that ChatGPT’s success had made salient. The Act that was adopted addresses the AI capabilities of 2023-2024; whether it will address the capabilities of 2027-2028 is uncertain.
The EU AI Act took approximately three years from the European Commission’s initial proposal to its final adoption. In those three years, the capabilities of AI systems changed dramatically — the initial proposal was written before ChatGPT existed, and significant revisions were required to address the general-purpose AI capabilities that ChatGPT’s success had made salient. The Act that was adopted addresses the AI capabilities of 2023-2024; whether it will address the capabilities of 2027-2028 is uncertain.
The speed problem is particularly acute for frontier AI systems, which are advancing rapidly. A regulatory framework designed around the capabilities of GPT-4 may be inadequate for systems with significantly more capability that will be deployed before the regulatory framework can be updated. The specific thresholds and requirements in the EU AI Act — including the computational threshold for frontier models — will need to be updated as capabilities advance, and the process for updating them is uncertain.
Several approaches to the speed problem have been proposed.
Adaptive regulation. Regulatory frameworks that include explicit mechanisms for updating requirements as technology changes — sunset clauses, mandatory reviews, delegated authority to update technical standards — can reduce the lag between technology and governance. The EU AI Act includes some adaptive elements, including delegated authority to update the list of prohibited applications and high-risk systems.
Principles-based regulation. Rather than specifying requirements at the level of specific technical details, principles-based regulation specifies higher-level outcomes — “AI systems must be safe for their intended use” — and allows the specific requirements to be developed by technical bodies closer to the technology. Principles-based regulation is more adaptable but less specific, and may be less enforceable.
Outcome-based accountability. Rather than regulating the design of AI systems, outcome-based accountability holds organisations responsible for the harms their systems cause, regardless of the specific design. Outcome-based accountability creates incentives for safety without prescribing specific technical approaches, and adapts automatically as the technology changes because it focuses on outcomes rather than means.
None of these approaches fully resolves the speed problem, but each addresses it in specific ways that are relevant to the design of AI governance frameworks.
The Frontier Model Forum: Industry Self-Governance
In July 2023, Anthropic, Google, Microsoft, and OpenAI announced the formation of the Frontier Model Forum — an industry body focused on “AI safety research for frontier models” and on “developing standards and best practices for the responsible development of frontier models.”
- Date:
- July 26, 2023
- Location:
- United States
- Significance:
- Four leading AI companies (Anthropic, Google, Microsoft, OpenAI) announced an industry self-governance body for frontier model safety — the most organised attempt at industry self-governance for frontier AI
- Outcome:
- Established a research fund and developed a common framework for evaluating dangerous capabilities; faced structural criticism that an industry body governed by its member companies cannot substitute for public governance
The Frontier Model Forum represents the most organised attempt at industry self-governance for frontier AI, and its formation and activities reveal both the potential and the limitations of industry self-governance as an approach to AI governance.
The Forum’s stated goals include: conducting and funding research on AI safety; identifying best practices for frontier model development; collaborating with policymakers and academics on AI safety; and developing technical standards and benchmarks for evaluating frontier AI safety.
In its first year of operation, the Forum established a research fund and supported several specific safety research initiatives. It developed a common framework for evaluating dangerous capabilities in frontier models, which provided the basis for the evaluations required by several regulatory frameworks.
The limitations of industry self-governance are structural. An industry body governed by its member companies has governance challenges analogous to those of any trade association: the interests of the members may conflict with the public interest, and the governance decisions of the body reflect the interests of those who control it. The Frontier Model Forum’s membership does not include the full range of AI developers, and its governance decisions do not have democratic accountability.
More fundamentally, industry self-governance cannot substitute for public governance in domains where the stakes are high and where the interests of the industry may diverge from the public interest. The pharmaceutical industry has self-regulatory bodies for drug safety; they are supplemented, not substituted, by public regulatory agencies like the FDA. The AI industry’s self-governance organisations are a useful complement to public governance but cannot replace it.
The Nuclear Analogy: How We Governed the Last Dangerous Technology
AI governance discussions frequently invoke the analogy to nuclear weapons — the last technology that was widely considered both transformative and potentially existential. The analogy is instructive both for what it suggests about the possibilities and for what it reveals about the differences.
The governance of nuclear weapons involved several elements that AI governance currently lacks:
International treaties with verification. The Nuclear Non-Proliferation Treaty (NPT) established a framework in which most countries agreed not to develop nuclear weapons, and nuclear-armed states agreed to work toward disarmament. The Comprehensive Nuclear-Test-Ban Treaty prohibited nuclear testing. The New START Treaty between the US and Russia established specific limits on nuclear arsenals with verification mechanisms.
An international agency with inspection authority. The International Atomic Energy Agency (IAEA) was established as an international technical agency with the authority to conduct inspections of nuclear facilities to verify treaty compliance. The IAEA’s inspection authority created a form of international accountability that no current AI governance body possesses.
Export controls. Export control regimes — coordinated through the Wassenaar Arrangement and other frameworks — restricted the transfer of nuclear-related technology and materials to countries without existing nuclear capabilities.
Common understanding of catastrophic risk. The nuclear governance framework was built on a common understanding — shared even by Cold War adversaries — that nuclear war would be catastrophic for all parties. This common understanding created a specific incentive for cooperation that other competitive dynamics did not undermine.
- Date:
- July 1, 1968 (signed); March 5, 1970 (entered into force)
- Location:
- United Nations, New York, USA
- Significance:
- The foundational international treaty governing nuclear weapons — most countries agreed not to develop them, and the five recognised nuclear-armed states agreed to work toward disarmament
- Outcome:
- Created the framework for half a century of nuclear non-proliferation policy; frequently invoked as the closest historical analogue for what international AI governance would require
The differences between nuclear and AI governance challenges are also significant:
AI capabilities are more broadly distributed than nuclear capabilities. The knowledge and equipment required to build nuclear weapons are highly specialised and can be controlled through export restrictions and facility monitoring. The knowledge and equipment required to develop advanced AI systems are much more broadly distributed — AI research is conducted in universities and companies worldwide, and the computing hardware is commercial technology.
AI systems can be replicated. A nuclear weapon is a physical object; once controlled, it stays controlled. An AI system is software; once trained, it can be copied and distributed infinitely. The enforcement of AI governance requires not just controlling physical objects but controlling information.
AI capabilities are dual-use in a more fundamental sense. Nuclear technology is primarily weapons technology, with a secondary civilian use in power generation. AI technology is primarily civilian technology, with a secondary weapons and surveillance application. The dual-use nature of AI makes controlling it more difficult than controlling nuclear technology.
Despite these differences, the nuclear analogy suggests that effective AI governance will require elements analogous to the nuclear governance framework: binding international agreements, technical agencies with inspection authority, export controls on the most sensitive capabilities, and a common understanding of the most serious risks.
The Governance Proposals: What the Research Community Suggests
The AI governance research community — political scientists, legal scholars, governance theorists, and AI researchers who study governance — has produced a range of specific proposals for addressing the governance gap. Understanding these proposals and their tradeoffs illuminates what governance that was adequate to the challenge might look like.
A global AI agency. Several researchers have proposed the creation of a Global AI Agency — an international organisation with authority over frontier AI development analogous to the IAEA’s authority over nuclear technology. The agency would conduct evaluations of frontier AI systems, establish safety standards, and have authority to require modifications or restrictions on systems that posed unacceptable risks.
The proposal faces significant political obstacles. The major AI powers — the United States and China — have incentives to avoid subjecting their AI development to international oversight. Building the technical capacity and political consensus for such an agency would require sustained diplomatic effort over years.
Compute governance. A more technically tractable approach to AI governance focuses on the computing resources required to train frontier AI systems. Because frontier AI training requires enormous quantities of specific computing hardware — primarily NVIDIA’s A100 and H100 GPUs — it is possible to monitor and regulate the development of frontier AI through the hardware supply chain.
Compute governance proposals include: export controls on the most powerful AI chips (already being implemented through US export controls on China); monitoring requirements for large-scale training runs; and licensing requirements for compute above specific thresholds.
Compute governance is more tractable than direct AI governance because it operates through hardware supply chains that are more controllable than the diffuse AI research ecosystem. But it has limitations: it only reaches the training of new models, not the deployment of existing ones; it can be evaded through distributed training across smaller clusters; and it gives the hardware supply chain control — primarily NVIDIA and the US government — an enormous amount of power over AI development globally.
Differential safety requirements. Rather than uniform requirements for all AI systems, differential safety requirements calibrate the regulatory burden to the capability and the risk profile of specific systems. This is essentially the EU AI Act’s approach — higher-risk systems face more stringent requirements — but proposals in this vein often suggest more specific capability thresholds and more stringent requirements for the highest-capability systems.
Democratic participation mechanisms. Proposals for increasing democratic participation in AI governance include requirements for community consultation in AI deployment decisions, participatory design processes for AI systems in public services, and democratic oversight boards for AI-using public institutions. These proposals address the democratic deficit but face implementation challenges in diverse democratic systems with different governance traditions.
What Good Governance Would Look Like
Drawing on the specific governance frameworks being developed, the proposals from the research community, and the lessons of previous technology governance experiences, it is possible to sketch what adequate AI governance would look like — not as a specific blueprint but as a set of principles and institutional elements.
Mandatory safety evaluation before deployment. Frontier AI systems should be subject to mandatory safety evaluation before deployment in any high-stakes application. The evaluation should be conducted by independent technical bodies — not self-reported by the developers — and should address the specific dangerous capabilities that researchers have identified as most concerning.
International technical standards. Interoperability between national governance frameworks requires common technical standards — agreed measures of AI system capability and risk, agreed evaluation methodologies, agreed disclosure requirements. These standards should be developed by technically capable international bodies with broad participation, including from the global south.
Binding international agreements on specific dangerous applications. Even if comprehensive international AI governance is not achievable in the near term, binding international agreements on specific dangerous applications — autonomous weapons that select and engage targets without human oversight, AI-enabled mass surveillance systems, AI used for biological weapons design — may be more achievable. Specific, bounded agreements are easier to negotiate and verify than comprehensive frameworks.
Adaptive governance mechanisms. Governance frameworks must include explicit mechanisms for updating requirements as AI capabilities evolve. Standing technical bodies with delegated authority to update specific standards, mandatory periodic reviews of framework adequacy, and sunset clauses for specific provisions that may become obsolete are all elements of adaptive governance.
Democratic accountability. Effective AI governance requires democratic accountability — mechanisms through which the people most affected by AI development can influence how it proceeds. The specific mechanisms will differ across different democratic systems, but the principle — that AI governance should be accountable to those who bear its consequences — is fundamental.
Global south inclusion. AI governance frameworks must include meaningful representation of the global south — the countries that will be most affected by AI systems developed primarily in the United States and China and that currently have the least influence over how those systems are developed. This requires both formal representation in governance institutions and capacity building to enable meaningful participation.
The Urgency: Why This Cannot Wait
The governance gap would be a manageable problem if AI development were proceeding slowly, allowing governance to catch up incrementally. AI development is not proceeding slowly.
The capabilities of frontier AI systems are advancing rapidly. The systems that exist today are significantly more capable than the systems of two years ago, and the systems of two years from now will be more capable still. The governance frameworks that are adequate for the AI systems of 2024 may be inadequate for the AI systems of 2026.
The deployment of AI systems is accelerating. The systems being deployed today are reaching hundreds of millions of users, affecting decisions in healthcare, in education, in employment, in criminal justice, and in political information. The harms that inadequate governance allows are not hypothetical — they are occurring at scale, now, with the AI systems that have already been deployed.
The governance frameworks that are being built will set precedents that will be difficult to revise. The institutional structures, the legal frameworks, the international norms that are established in the current period of AI development will shape the governance environment for decades. Getting them right now matters more than getting them right later, because the cost of revision increases as the technology and the governance framework both mature.
And the window for establishing effective governance may be closing. As AI systems become more capable, as the economic interests invested in specific development trajectories become larger, as the geopolitical competition between major AI powers becomes more entrenched — the political feasibility of significant governance changes may decrease rather than increase.
The urgency is real. The governance gap is large. The question is whether the political will exists to close it at the pace that the situation requires.
The Honest Assessment: How Much Time We Have
The honest assessment of the governance gap is uncomfortable. The governance frameworks being built are more advanced than they were five years ago but are still significantly inadequate to the challenge of governing AI development over the next decade.
The EU AI Act is a significant achievement, but it is one jurisdiction’s framework for one set of AI applications. The US AI Safety Institute is a meaningful institutional development, but it lacks enforcement authority. The Bletchley Summit produced a meaningful international statement, but not binding commitments. The gap between what exists and what is needed is not closing as fast as the technology is advancing.
The EU AI Act is a significant achievement, but it is one jurisdiction’s framework for one set of AI applications. The US AI Safety Institute is a meaningful institutional development, but it lacks enforcement authority. The Bletchley Summit produced a meaningful international statement, but not binding commitments. The voluntary commitments of major AI companies reflect genuine engagement with safety, but voluntary commitments are insufficient for the most consequential decisions.
The gap between what exists and what is needed is not closing as fast as the technology is advancing. The competitive dynamics of AI development — the race between organisations and nations for AI capability — are creating pressure against the kind of coordinated, cautious approach that adequate safety would require.
Whether the governance gap will be closed before the technology advances beyond what current governance can manage is the most important open question in AI policy. The answer depends on political decisions that are still being made — by governments, by international institutions, by the AI companies themselves, and by the citizens and civil society organisations that are trying to hold those decisions accountable.
The diplomat from the Pacific island nation is right: the people most affected by AI development are not the people who are shaping it. Changing that — building the governance frameworks that make AI development genuinely accountable to the full range of people it will affect — is the work that the current period demands.
Whether humanity can govern what it has built is not yet determined. That it must try is not in doubt.
Whether humanity can govern what it has built is not yet determined. That it must try is not in doubt.
Further Reading
- “The AI Act” — European Parliament documentation — The full text and official summaries of the EU AI Act, the most significant AI governance framework yet enacted.
- “Governing AI: A Blueprint for the Future” — various policy institutes — The Brookings Institution, the Council on Foreign Relations, and similar organisations have produced comprehensive analyses of AI governance options.
- “The Geopolitics of AI” by various authors — Multiple books and reports on the US-China AI competition and its governance implications, including analyses from the Stimson Center, the Georgetown Center for Security and Emerging Technology, and similar organisations.
- “AI Governance: A Research Agenda” by various academic authors — Academic analyses of AI governance challenges and proposals, published in policy journals including Global Policy, Science and Public Policy, and AI & Society.
- Bletchley Park Declaration (2023) and subsequent international statements — The primary international governance documents from the AI Safety Summits, available from the UK government.
The most important open questions in AI research — about architecture, about training, about generalisation, about alignment, about consciousness — and the research programmes that are trying to answer them. The frontier of AI science, as seen from 2025.
Comments
Reply on Bluesky → (opens in a new tab)