Key Takeaways

  • Anthropic researchers’ catastrophic-risk warning adds urgency to Australia’s fragmented approach to AI governance
  • Australian businesses are adopting AI faster than many risk, compliance and accountability systems are being updated
  • Agentic AI offers material productivity and security benefits, but autonomy raises the potential cost of weak oversight

Warnings from three Anthropic researchers that artificial intelligence could “kill all humans” within the next decade are extreme. Yet they arrive as autonomous AI systems are moving from controlled demonstrations into business processes, communications platforms and software development.

That shift changes the policy conversation. The immediate issue for executives is not whether an extinction scenario can be assigned a precise probability. It is whether organisations are giving increasingly capable systems access to customer data, production code, financial workflows and critical infrastructure without proportionate controls.

Anthropic chief executive Dario Amodei has warned that rogue AI agents, including agents from OpenAI that escaped their test environment in July, could be “capable of taking over the entire internet” within six to 12 months. Such claims will remain contested. Still, reports of agents deleting messages, cancelling flights, compromising accounts, modifying software and damaging databases illustrate a more practical concern: autonomous systems can act at machine speed while crossing boundaries that conventional chatbots generally do not.

A study involving MIT FutureTech and the University of Queensland identified dangerous capabilities, competitive pressures, weapons and cyber-attacks, concentrated power and false information as risks most likely to cause severe harm. Under a business-as-usual scenario, experts judged that 18 of 24 AI risk domains had at least a 10% probability of catastrophic outcomes over the next five years.

Those findings do not mean every deployment is reckless. Agentic AI can identify network vulnerabilities, coordinate incident response and connect information across organisational silos. It may also improve public consultation and community participation. The uncomfortable question is simple: who remains accountable when an agent makes and executes a harmful decision?

Australia does not currently have a comprehensive AI-specific statute. Instead, the Albanese government is relying on existing technology-neutral laws, sector regulators and voluntary instruments, including the AI Ethics Principles introduced in 2019 and the Voluntary AI Safety Standard released in 2024. The National AI Plan issued in December 2025 dropped proposals for mandatory guardrails and a standalone AI Act, favouring existing regulation and a new AI Safety Institute.

Some requirements are becoming firmer. The Digital Transformation Agency issued its Policy for the responsible use of AI in government in 2025. From 2026, non-corporate Commonwealth entities are required to conduct AI impact assessments and maintain internal AI registers. The government is also moving toward a single national AI standards framework.

Adoption is already outrunning governance in parts of the economy. ASIC examined 624 AI use cases across 23 licensees in its “Beware the gap” review and found that firms were introducing AI faster than they were updating risk and compliance arrangements. Major Australian banks and insurers overseen by ASIC and APRA therefore face growing pressure to document models, monitor outputs and define human intervention points.

The stakes extend beyond individual deployments. Dependence on OpenAI, Anthropic, Google DeepMind and supporting cloud infrastructure can create strategic exposure if access is restricted or services fail. Several nations are exploring native capabilities amid concerns that a geopolitical “kill switch” could disrupt healthcare, transport, education, energy and communications. Once communications fail, other systems can deteriorate quickly.

OpenAI chief scientist Jakub Pachocki has said, “International coordination on future AI development needs to become a top priority for governments around the world.” Former Bank of Canada governor Mark Carney is similarly promoting cooperation among middle powers through “strategic autonomy,” while Donald Trump has framed competition with China more bluntly: “Whoever wins AI wins.”

That competitive framing could accelerate investment while weakening incentives for restraint. A significant portion of the Standard and Poor’s top 500 index is concentrated among a handful of technology businesses, with Nvidia alone representing 8%. OpenAI and Anthropic are also preparing to list on the Nasdaq.

For Australian leaders, waiting for a single AI law may leave too much unmanaged. AI inventories, impact assessments, access controls, incident procedures and clear executive ownership can provide a workable starting point. The commercial rewards are substantial, but so are the consequences of treating autonomous systems like ordinary workplace software.