In AI and nuclear alike, extraordinary claims need extraordinary evidence
In 1942, scientists working on what became the Manhattan Project confronted a disturbing possibility: could an atomic explosion become hot enough to ignite a self-sustaining nuclear reaction in the atmosphere?
They took the question seriously and did the physics. Hans Bethe, Emil Konopinski and others examined whether the reactions could release energy faster than the atmosphere could dissipate it, and their calculations showed they could not. Three years later, the Trinity test went ahead.
People working in AI tend to bristle at nuclear comparisons, and the objection is not unreasonable. Every sector regards its own technology as unique, and AI differs from the bomb in obvious ways: it is software rather than a physical device, it spreads through open publication rather than fissile material and its developers are private companies rather than wartime state programs. But the comparison here is about the discipline the scientists applied to a hypothesis they couldn’t yet rule out. That discipline transfers regardless of the technology involved. The scientists developing the atomic bomb were developing something whose limits they were still discovering, in the middle of a war, initially fearing that Nazi Germany was pursuing the same weapon. They investigated an existential hypothesis until the evidence could bear the weight of the decision.
In a 12 September essay, Anthropic chief executive Dario Amodei called for the AI industry to ‘pace the frontier’. Among his concerns was the possibility that within six to 12 months a swarm of AI agents could take over much of the internet through a persistent botnet, a network of compromised computers acting under a single controller
Such possibilities deserve serious investigation, particularly when raised by people with access to the world’s most capable models. But they are also extraordinary claims, and extraordinary claims require extraordinary evidence. Most people cannot independently assess frontier AI capabilities and must trust those building them, which gives technical experts a particular responsibility when they talk about catastrophic risk.
AI risk arguments tend to begin with demonstrated capabilities and extrapolate from them. A model shows impressive offensive cyber capability, researchers envisage autonomous agents chaining those capabilities together, and scaling that across millions of systems produces an AI-controlled botnet. While each link may be technically plausible, establishing the outcome’s probability takes considerably more work than establishing that the chain exists. National security analysts should recognise the distinction. Threat modelling explores what could happen; good analysis then examines demonstrated capability, likelihood and consequence separately. Policy suffers when those categories collapse into one another.
The distinction matters all the more because AI development is taking place amid strategic competition with China. Amodei recognises the tension. He argues that democracies need a sufficiently large lead over China to give themselves room to slow down, while acknowledging that a comprehensive agreement is unlikely soon because the incentive to cheat would be enormous. That makes pacing a difficult control problem: restrictions adopted in California can be inspected in California, whereas knowing whether equivalent restrictions are being observed in China is considerably harder. Here the nuclear analogy earns its keep. The Manhattan Project scientists confronted a catastrophic hypothesis while racing an adversary, and their response was to investigate the risk and engineer against what they found.
The AI debate needs the same discipline, not least because existential framing distorts the arithmetic. Once human extinction appears on one side of a risk calculation, almost any cost on the other side becomes tolerable. But the severity of an outcome tells us little about its probability, and probability should be doing the work.
If frontier models really are approaching the ability to compromise the internet autonomously, we should demand evidence commensurate with that claim: the experiment; the capabilities observed; and the assumptions connecting those observations to the predicted outcome, with demonstrated behaviour distinguished from extrapolation. We should put considerably more effort into engineering the controls. Amodei himself identifies many of them, including monitoring, sandboxing, operational security, evaluation and interpretability. However new the technology, these are familiar engineering problems. Solving them may prove more durable than any attempt to regulate the speed of progress itself.
There is also a simpler option for the companies raising the alarm. If Anthropic believes its unreleased models are approaching an unsafe threshold, it can slow their development. OpenAI can do the same. They possess evidence the rest of us cannot see, and acting on it themselves would be a meaningful signal.
Government intervention carries broader consequences. Regulation can entrench incumbents and constrain companies well behind the frontier, while open-weight models and cheaper hardware mean capability will diffuse beyond the laboratories governments can readily supervise. That should push policymakers towards controls that remain useful as the frontier moves: hardening critical infrastructure; securing model weights; constraining what autonomous systems can access and execute; and setting measurable thresholds that trigger stronger safeguards as evidence accumulates.
Perhaps the evidence will eventually tell us that frontier development must slow. Amodei may already be seeing capabilities that should concern us greatly. But Los Alamos offers the better standard. Its scientists confronted an extraordinary possibility, calculated the risk and engineered around what the evidence showed, all while operating in the strategic environment they actually inhabited. Catastrophic risks deserve to be taken seriously, which means doing the work, exposing the assumptions and building the controls. The ability to imagine a catastrophe cannot carry the same weight as evidence that one is coming.
About Author
