The temptation to automate command and control will be great. The danger is greater.

No technology since the atomic bomb has inspired the apocalyptic imagination like artificial intelligence. Ever since ChatGPT began exhibiting glints of logical reasoning in November, the internet has been awash in doomsday scenarios. Many are self-consciously fanciful—they’re meant to jar us into envisioning how badly things could go wrong if an emerging intelligence comes to understand the world, and its own goals, even a little differently from how its human creators do. One scenario, however, requires less imagination, because the first steps toward it are arguably already being taken—the gradual integration of AI into the most destructive technologies we possess today.

The world’s major military powers have begun a race to wire AI into warfare. For the moment, that mostly means giving algorithms control over individual weapons or drone swarms. No one is inviting AI to formulate grand strategy, or join a meeting of the Joint Chiefs of Staff. But the same seductive logic that accelerated the nuclear arms race could, over a period of years, propel AI up the chain of command. How fast depends, in part, on how fast the technology advances, and it appears to be advancing quickly. How far depends on our foresight as humans, and on our ability to act with collective restraint.

Jacquelyn Schneider, the director of the Wargaming and Crisis Simulation Initiative at Stanford’s Hoover Institution, recently told me about a game she devised in 2018. It models a fast-unfolding nuclear conflict and has been played 115 times by the kinds of people whose responses are of supreme interest: former heads of state, foreign ministers, senior NATO officers. Because nuclear brinkmanship has thankfully been historically rare, Schneider’s game gives us one of the clearest glimpses into the decisions that people might make in situations with the highest imaginable human stakes.

It goes something like this: The U.S. president and his Cabinet have just been hustled into the basement of the West Wing to receive a dire briefing. A territorial conflict has turned hot, and the enemy is mulling a nuclear first strike against the United States. The atmosphere in the Situation Room is charged. The hawks advise immediate preparations for a retaliatory strike, but the Cabinet soon learns of a disturbing wrinkle. The enemy has developed a new cyberweapon, and fresh intelligence suggests that it can penetrate the communication system that connects the president to his nuclear forces. Any launch commands that he sends may not reach the officers responsible for carrying them out.

There are no good options in this scenario. Some players delegate launch authority to officers at missile sites, who must make their own judgments about whether a nuclear counterstrike is warranted—a scary proposition. But Schneider told me she was most unsettled by a different strategy, pursued with surprising regularity. In many games, she said, players who feared a total breakdown of command and control wanted to automate their nuclear launch capability completely. They advocated the empowerment of algorithms to determine when a nuclear counterstrike was appropriate. AI alone would decide whether to enter into a nuclear exchange.

Schneider’s game is, by design, short and stressful. Players’ automation directives were not typically spelled out with an engineer’s precision—how exactly would this be done? Could any automated system even be put in place before the culmination of the crisis?—but the impulse is telling nonetheless. “There is a wishful thinking about this technology,” Schneider said, “and my concern is that there will be this desire to use AI to decrease uncertainty by [leaders] who don’t understand the uncertainty of the algorithms themselves.”

AI offers an illusion of cool exactitude, especially in comparison to error-prone, potentially unstable humans. But today’s most advanced AIs are black boxes; we don’t entirely understand how they work. In complex, high-stakes adversarial situations, AI’s notions about what constitutes winning may be impenetrable, if not altogether alien. At the deepest, most important level, an AI may not understand what Ronald Reagan and Mikhail Gorbachev meant when they said, “A nuclear war cannot be won.”

there is precedent, of course, for the automation of Armageddon. After the United States and the Soviet Union emerged as victors of the Second World War, they looked set to take up arms in a third, a fate they avoided only by building an infrastructure of mutual assured destruction. This system rests on an elegant and terrifying symmetry, but it goes wobbly each time either side makes a new technological advance. In the latter decades of the Cold War, Soviet leaders worried that their ability to counter an American nuclear strike on Moscow could be compromised, so they developed a “dead hand” program.

It was so simple, it barely qualified as algorithmic: Once activated during a nuclear crisis, if a command-and-control center outside Moscow stopped receiving communications from the Kremlin, a special machine would inquire into the atmospheric conditions above the capital. If it detected telltale blinding flashes and surges in radioactivity, all the remaining Soviet missiles would be launched at the United States. Russia is cagey about this system, but in 2011, the commander of the country’s Strategic Missile Forces said it still exists and is on “combat duty.” In 2018, a former leader of the missile forces said it has “even been improved.”

In 2019, Curtis McGiffin, an associate dean at the Air Force Institute of Technology, and Adam Lowther, then the director of research and education at the Louisiana Tech Research Institute, published an article arguing that America should develop its own nuclear dead hand. New technologies have shrunk the period of time between the moment an incoming attack is detected and the last moment that a president can order a retaliatory salvo. If this decision window shrinks any further, America’s counterstrike ability could be compromised. Their solution: Backstop America’s nuclear deterrent with an AI that can make launch decisions at the speed of computation.

McGiffin and Lowther are right about the decision window. During the early Cold War, bomber planes like the one used over Hiroshima were the preferred mode of first strike. These planes took a long time to fly between the Soviet Union and the United States, and because they were piloted by human beings, they could be recalled. Americans built an arc of radar stations across the Canadian High Arctic, Greenland, and Iceland so that the president would have an hour or more of warning before the first mushroom cloud bloomed over an American city. That’s enough time to communicate with the Kremlin, enough time to try to shoot the bombers down, and, failing that, enough time to order a full-scale response.

The intercontinental ballistic missile (ICBM), first deployed by the Soviet Union in 1958, shortened that window, and within a decade, hundreds of them were slotted into the bedrock of North America and Eurasia. Any one of them can fly across the Northern Hemisphere in less than 30 minutes. To preserve as many of those minutes as possible, both superpowers sent up fleets of satellites that could spot the unique infrared signature of a missile launch in order to grok its precise parabolic path and target.

After nuclear-armed submarines were refined in the ’70s, hundreds more missiles topped with warheads began to roam the world’s oceans, nearer to their targets, cutting the decision window in half, to 15 minutes or perhaps fewer. (Imagine one bobbing up along the Delaware coast, just 180 miles from the White House.) Even if the major nuclear powers never successfully develop new nuclear-missile technology, 15 minutes or fewer is frighteningly little time for a considered human response. But they are working to develop new missile technology, including hypersonic missiles, which Russia is already using in Ukraine to strike quickly and evade missile defenses. Both Russia and China want hypersonic missiles to eventually carry nuclear warheads. These technologies could potentially cut the window in half again.

These few remaining minutes would go quickly, especially if the Pentagon couldn’t immediately conclude that a missile was headed for the White House. The president may need to be roused from sleep; launch codes could be fumbled. A decapitation strike could be completed with no retaliatory salvo yet ordered. Somewhere outside D.C., command and control would scramble to find the next civilian leader down the chain, as a more comprehensive volley of missiles rained down upon America’s missile silos, its military bases, and its major nodes of infrastructure.

A first strike of this sort would still be mad to attempt, because some American nuclear forces would most likely survive the first wave, especially submarines. But as we have learned again in recent years, reckless people sometimes lead nuclear powers. Even if the narrowing of the decision window makes decapitation attacks only marginally more tempting, countries may wish to backstop their deterrent with a dead hand.

the united states is not yet one of those countries. After McGiffin and Lowther’s article was published, Lieutenant General John Shanahan, the director of the Pentagon’s Joint Artificial Intelligence Center, was asked about automation and nuclear weapons. Shanahan said that although he could think of no stronger proponent for AI in the military than himself, nuclear command and control “is the one area I pause.”

The Pentagon has otherwise been working fast to automate America’s war machine. As of 2021, according to a report that year, it had at least 685 ongoing AI projects, and since then it has continually sought increased AI funding. Not all of the projects are known, but a partial vision of America’s automated forces is coming into view. The tanks that lead U.S. ground forces in the future will scan for threats on their own so that operators can simply touch highlighted spots on a screen to wipe out potential attackers. In the F-16s that streak overhead, pilots will be joined in the cockpit by algorithms that handle complex dogfighting maneuvers. Pilots will be free to focus on firing weapons and coordinating with swarms of autonomous drones.

In January, the Pentagon updated its previously murky policy to clarify that it will allow the development of AI weapons that can make kill shots on their own. This capability alone raises significant moral questions, but even these AIs will be operating, essentially, as troops. The role of AI in battlefield command and the strategic functioning of the U.S. military is largely limited to intelligence algorithms, which simultaneously distill data streams gathered from hundreds of sensors—underwater microphones, ground radar stations, spy satellites. AI won’t be asked to control troop movements or launch coordinated attacks in the very near future. The pace and complexity of warfare may increase, however, in part because of AI weapons. If America’s generals find themselves overmatched by Chinese AIs that can comprehend dynamic, million-variable strategic situations for weeks on end, without so much as a nap—or if the Pentagon fears that could happen—AIs might be placed in higher decision-making roles.

The precise makeup of America’s nuclear command and control is classified, but AI’s awesome processing powers are already being put to good use in the country’s early-alert systems. Even here, automation presents serious risks. In 1983, a Soviet early-alert system mistook glittering clouds above the Midwest for launched missiles. Catastrophe was averted only because Lieutenant Colonel Stanislav Petrov—a man for whom statues should be raised—felt in his gut that it was a false alarm. Today’s computer-vision algorithms are more sophisticated, but their workings are often mysterious. In 2018, AI researchers demonstrated that tiny perturbations in images of animals could fool neural networks into misclassifying a panda as a gibbon. If AIs encounter novel atmospheric phenomena that weren’t included in their training data, they may hallucinate incoming attacks.

But put hallucinations aside for a moment. As large language models continue to improve, they may eventually be asked to generate lucid text narratives of fast-unfolding crises in real time, up to and including nuclear crises. Once these narratives move beyond simple statements about the number and location of approaching missiles, they will become more like the statements of advisers, engaged in interpretation and persuasion. AIs may prove excellent advisers—dispassionate, hyperinformed, always reliable. We should hope so, because even if they are never asked to recommend responses, their stylistic shadings would undoubtedly influence a president.

Given wide enough leeway over conventional warfare, an AI with no nuclear-weapons authority could nonetheless pursue a gambit that inadvertently escalates a conflict so far and so fast that a panicked nuclear launch follows. Or it could purposely engineer battlefield situations that lead to a launch, if it thinks the use of nuclear weapons would accomplish its assigned goals. An AI commander will be creative and unpredictable: A simple one designed by OpenAI beat human players at a modified version of Dota 2, a battle simulation game, with strategies that they’d never considered. (Notably, it proved willing to sacrifice its own fighters.)

These more far-flung scenarios are not imminent. AI is viewed with suspicion today, and if its expanding use leads to a stock-market crash or some other crisis, these possibilities will recede, at least for a time. But suppose that, after some early hiccups, AI instead performs well for a decade or several decades. With that track record, it could perhaps be allowed to operate nuclear command and control in a moment of crisis, as envisioned by Schneider’s war-game participants. At some point, a president might preload command-and-control algorithms on his first day in office, perhaps even giving an AI license to improvise, based on its own impressions of an unfolding attack.

Much would depend on how an AI understands its goals in the context of a nuclear standoff. Researchers who have trained AI to play various games have repeatedly encountered a version of this problem: An AI’s sense of what constitutes victory can be elusive. In some games, AIs have performed in a predictable manner until some small change in their environment caused them to suddenly shift their strategy. For instance, an AI was taught to play a game where players look for keys to unlock treasure chests and secure a reward. It did just that until the engineers tweaked the game environment, so that there were more keys than chests, after which it started hoarding all the keys, even though many were useless, and only sometimes trying to unlock the chests. Any innovations in nuclear weapons—or defenses—could lead an AI to a similarly dramatic pivot.

Any country that inserts AI into its command and control will motivate others to follow suit, if only to maintain a credible deterrent. Michael Klare, a peace-and-world-security-studies professor at Hampshire College, has warned that if multiple countries automate launch decisions, there could be a “flash war” analogous to a Wall Street “flash crash.” Imagine that an American AI misinterprets acoustic surveillance of submarines in the South China Sea as movements presaging a nuclear attack. Its counterstrike preparations would be noticed by China’s own AI, which would actually begin to ready its launch platforms, setting off a series of escalations that would culminate in a major nuclear exchange.

in the early ’90s, during a moment of relative peace, George H. W. Bush and Mikhail Gorbachev realized that competitive weapons development would lead to endlessly proliferating nuclear warheads. To their great credit, they refused to submit to this arms-race dynamic. They instead signed the Strategic Arms Reduction Treaty, the first in an extraordinary sequence of agreements that shrank the two countries’ arsenals to less than a quarter of their previous size.

History has since resumed. Some of those treaties expired. Others were diluted as relations between the U.S. and Russia cooled. The two countries are now closer to outright war than they have been in generations. On February 21 of this year, less than 24 hours after President Joe Biden strolled the streets of Kyiv, Russian President Vladimir Putin said that his country would suspend its participation in New START, the last arsenal-limiting treaty that remains in effect. Meanwhile, China now likely has enough missiles to destroy every major American city, and its generals have reportedly grown fonder of their arsenal as they have seen the leverage that nuclear weapons have afforded Russia during the Ukraine war. Mutual assured destruction is now a three-body problem, and every party to it is pursuing technologies that could destabilize its logic.

The next moment of relative peace could be a long way away, but if it comes again, we should draw inspiration from Bush and Gorbachev. Their disarmament treaties were ingenious because they represented a recovery of human agency, as would a global agreement to forever keep AI out of nuclear command and control. Some of the scenarios set forth here may sound quite distant, but that’s more reason to think about how we can avoid them, before AI reels off an impressive run of battlefield successes and its use becomes too tempting.

A treaty can always be broken, and compliance with this one would be particularly difficult to verify, because AI development doesn’t require conspicuous missile silos or uranium-enrichment facilities. But a treaty can help establish a strong taboo, and in this realm a strongly held taboo may be the best we can hope for. We cannot encrust the Earth’s surface with automated nuclear arsenals that put us one glitch away from apocalypse. If errors are to deliver us into nuclear war, let them be our errors. To cede the gravest of all decisions to the dynamics of technology would be the ultimate abdication of human choice.