Key Takeaways

  • Control AI is backing legislation that would ban the creation of artificial superintelligence in the UK.
  • The alignment science lead at Anthropic placed the chance of AI killing all humans within the next decade above 10%.
  • The dispute raises practical questions for businesses about independent testing, model access, governance, and international coordination.

Warnings about artificial superintelligence have moved from research circles into mainstream politics, with tech employees, UK parliamentarians, and US lawmakers intensifying calls for stronger oversight of advanced AI development.

At a Westminster session convened by Control AI, a former defence secretary stated that superintelligent AI could pose a threat on the same scale as nuclear weapons, or possibly a greater one. A Berkeley computer scientist raised the prospect of a catastrophic failure, warning that a severe incident could leave humanity without meaningful control.

Control AI is backing a bill tabled by a UK Member of Parliament that seeks to ban the creation of artificial superintelligence, or ASI. The 2017 Nobel peace prize winner for leading the International Campaign to Abolish Nuclear Weapons noted to parliamentarians that humanity has confronted potentially species-ending capabilities before.

The alignment science lead at Anthropic stated that there was a greater than 10% chance AI could cause human extinction within the next decade. The lead also indicated Anthropic lacked a comprehensive plan for ensuring that ASI would remain aligned with human interests.

These remarks followed the resignation of a former Anthropic researcher who previously worked at OpenAI. The researcher stated neither company was acting responsibly, describing AI developers as gambling with human lives. The researcher argued that Anthropic understood the risks but remained caught in a race to reach more capable systems first.

Concern about catastrophic outcomes extends beyond campaigners. A 2024 survey of 2,778 researchers, summarized by Simple Research AI, found that between 38% and 51% assigned at least a 10% probability to outcomes from advanced AI as severe as human extinction (source). The median extinction estimate was about 5%.

Forecasting groups report differing estimates. A 2026 assessment from the Forecasting Research Institute reported a median expert estimate of 6% for human extinction by 2100, compared with 1% among superforecasters. This spread illustrates substantial uncertainty about technical progress and society's ability to manage it.

Enterprise buyers do not need to resolve the extinction debate before improving oversight; the immediate governance questions are concrete. Organizations must determine who evaluates a model before release, what evidence customers can inspect, and whether developers can restrict autonomous access to networks, code repositories, financial systems, or physical infrastructure. Organizations must also prepare for scenarios where a supplier declines external testing.

That final issue has become politically sensitive. According to the Financial Times, UK officials were concerned that Anthropic declined to submit an advanced model to the UK's AI Security Institute for pre-release testing. The Cabinet Office confirmed that only a few US organizations had received access, while stating the institute continued to collaborate with Anthropic.

For corporate boards, voluntary access arrangements can create an assurance gap. Internal safety teams at Anthropic, OpenAI, and Google DeepMind conduct alignment and catastrophic-risk research, but customers and governments seek independent evidence. The NIST AI Risk Management Framework offers one established starting point through its GOVERN, MAP, MEASURE, and MANAGE functions. The EU AI Act adds a legally oriented, risk-based structure.

A UK Member of Parliament has called for a multinational treaty governing superintelligence, while the chief scientist at OpenAI stated international coordination on future AI development needs to become a top government priority. A US senator is pressing Congress to regulate AI, citing polling that 81% of Americans want political action (source).

Not everyone accepts the catastrophic framing. A researcher at the Surrey Institute for People-Centred AI suggested advanced systems may prove too expensive and insufficiently useful. An academic at the Oxford Internet Institute argued that environmental costs, misinformation, and job displacement are more urgent than existential scenarios. A representative of Sofair noted that while AI is highly useful, unconstrained access to online services and critical systems presents immediate operational issues.

Businesses face present-day operational risks alongside uncertain future harms. Independent evaluation, staged deployment, access controls, and documented shutdown procedures offer concrete measures to begin securing enterprise environments, regardless of where executives place the odds of global catastrophe.