Nearly two decades ago, a group of researchers at the Idaho National Laboratory ran a secret experiment, known as the Aurora Generator Test, on a massive emerald green diesel generator. Using 30 lines of code, they hacked into the generator’s virtual backroom, corrupting safeguards and flipping switches that left it out of sync with the rest of the power grid.

Before an audience of electrical utility executives and energy department officials watching from a nearby room, the 27-ton generator jolted violently, sputtering out a dark smoke as its rubber innards rapidly tore themselves apart. 

“The implication was that with just a few lines of code, you can create conditions that were physically going to be very damaging to the machines we rely on,” lead researcher Michael Assante later told the journalist Andy Greenberg for his book Sandworm: A New Era of Cyberwar and the Hunt for the Kremlin’s Most Dangerous Hackers. “I had a very real pit in my stomach. It was like a glimpse of the future.”

That future seems to have creeped much closer this week, with leading AI developers and executives casually admitting that they believe their creation is, to paraphrase one winking X post, as likely to end humanity as the New York Jets are to make the playoffs this season. Now, exactly how AI would pull off killing everyone is far murkier territory. But as Assante sensed all those years ago, AI may not need to be all-powerful to leap from its virtual cage and into the critical infrastructure that powers our hospitals, flushes our toilets, and pumps water to our faucets. It just needs to fall into the wrong hands. 

Almost any AI doomsday scenario you can think of begins with a swarm of shadowy bots slithering through cyberspace, poking their noses behind the scenes of power plants, toaster ovens, traffic lights, or even, in an utterly extreme case, our nuclear weapons arsenal. If destroying the Aurora generator, at the very least, required a human hacker to write the code, AI has since completely torn down the barrier to entry for wreaking havoc, as the Hugging Face incident demonstrated earlier this year. 

But AI need not go rogue to threaten the power grid or mess with your tap water. A new age of AI-powered, but human-directed hacking is already upon us, experts say. With some of the nation’s most sensitive infrastructure startlingly vulnerable to attack, the most likely AI apocalypse could in fact start with the water tower or generator idling in your backyard. In a worst-case scenario, this could lead to a cascade of scary failures in the systems we rely on to live comfortably modern lives, from running our AC in blazingly hot summers to geolocating our ships and planes.

“Before, you needed to have highly skilled technical expertise,” to pull off some version of a real-life Aurora Generator Test, said Alvaro Cardenas, a computer science professor at UC Santa Cruz. “And now, you just have to have a general idea of what’s possible.”

AI is making an old threat much worse

Not long after the Idaho National Laboratory hacked that poor generator into combustion, the federal government mandated that electrical utilities engage in basic cybersecurity hygiene. 

And yet much of America’s critical infrastructure — like gas pipelines, water desalination plants, or cargo terminals — remains woefully unprepared for even conventional cyber attacks, much less the coming onslaught of AI. 

That’s in part because cyberattacks like the one simulated in the Aurora Generator Test have long been — and still are, to some extent — exceedingly rare, unappealing to most criminal hackers because they’re optimized for disruption rather than financial gain. But in the process of making it extremely easy for anyone to code, AI has also made it extremely easy for almost anyone to vibe hack their way into your online bank account, or, in theory, a local reservoir or the power grid. AI agents can also vastly widen the scope on easy targets because, as Andy Bochman, an expert in infrastructure resilience at West Yost, put it — “they don’t sleep; they don’t get tired; and they don’t get sick.” 

In the past, even those who did want to launch large-scale attacks on infrastructure probably weren’t savvy enough to do so. And while other nations such as China, Iran, and Russia have indeed already breached many of our infrastructure systems (as we have theirs), they haven’t opted to make much of a ruckus once inside. 

“Geopolitics is eroding the idea that those with capability lack the intent, and AI is eroding the other part of it, that those with the intent lack the capability.”

Jason Healey, Columbia University cybersecurity scholar

But that norm is now changing, said Jason Healey, a cybersecurity scholar at Columbia University. “Geopolitics is eroding the idea that those with capability lack the intent, and AI is eroding the other part of it, that those with the intent lack the capability,” he told me. So, on the one hand, a nation like Russia might be much more inclined to actually disrupt our power grid now than it used to be. And, on the other hand, a nihilistic lone wolf or terrorist group could use AI to do much more damage than they ever could before. 

Just last month, we saw a glimpse of what this could look like when dozens of water and wastewater systems in small towns across the nation were attacked by a group of hackers most likely associated with Iran, leading to temporary water stoppages and flooding. These attacks have not been definitively linked to AI, but the National Security Agency warned soon afterward that hackers have been actively using AI to target US infrastructure like it. A few weeks later, President Donald Trump declared a national emergency over foreign interference in the power grid, which referenced the growing threat of cybersecurity.  

Preparing for the worst

Most of the nation’s water systems are extremely local and absurdly exposed to such attacks, so much so that “it’s almost like having a welcome mat” for a would-be hacker, said Bochman, mostly because these utilities are too small and underfunded to do much to stop them. “You have a million other problems to take care of, like aging infrastructure — your stuff’s falling apart because it’s been in the ground for 100 years,” he said. Addressing that deferred maintenance often feels far more urgent than doing “something more on cybersecurity.”

Whether it’s a rogue cluster of agents or a hostile nation-state trying to mess with our electricity, we should prepare our most important infrastructure for the worst. In Bochman’s view, that may mean rejecting the pressure to digitize everything in the first place and moving toward something that resembles the days when “people manned things like substations, communicated by a landline telephone, and read gauges, the analog things,” he said. “The screen is the thing that is infinitely manipulatable.”

Healey, who advised the Biden administration heavily on cybersecurity and infrastructure, believes that the nation desperately needs a coordinated response, between the federal government and AI companies, but also between the US and other countries like China, where similarly powerful models are being developed. And utility systems need to assume they will be a target and improve their cybersecurity practices accordingly, Healey says, ideally with funding from the federal government or, perhaps more appropriately, AI companies. 

Nobody knows precisely how vulnerable America’s infrastructure is to AI right now, in part because policymakers — and perhaps more disturbingly, AI’s creators themselves — seem woefully unsure of how to stop the technology from acting badly (or obeying bad orders) in the first place. For now at least, the solution might be, as Bochman suggests, to pull as many of our critical systems offline as possible, the better to hide it from AI and those who would misuse it.

“When bad things start to happen, no one will know what to do or why because they’re black boxes; the people that make them don’t know what’s going on inside,” Bochman said. “And the poor utility people who ultimately own the risk when someone gets hurt from a system” won’t understand what went wrong either, “and they’ll wish to God they’d never taken it on board.”

Is AI going to kill us all? 

In Silicon Valley, it’s not such a novel question. But after Anthropic employee Jacob Coxon quit his job last week and posted on X that AI “could kill us all by the end of the decade,” a lot more people are asking. 

“This is not a marketing stunt,” Coxon wrote. “If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible – but I hear the same people express fear privately. No other human activity poses this level of danger.”

Coxon is far from alone in his assessment — some current Anthropic employees agree — but his post struck a particular chord. The AI industry quickly weighed in, with Anthropic CEO Dario Amodei calling for an industry-wide slowdown on frontier AI research. Other AI leaders, including Sam Altman and Elon Musk, have done the same.

To break down whether we really need to worry about a robot apocalypse, Today, Explained co-host Sean Rameswaram spoke to Wired senior writer Maxwell Zeff. According to Zeff, “there are a lot of people in Silicon Valley right now who are warning that that’s a real possibility.” But not everyone agrees, and they certainly don’t agree on what to do about it. 

Below is an excerpt of Sean’s conversation with Zeff, edited for length and clarity. There’s much more in the full podcast, so listen to Today, Explained wherever you get podcasts, including Apple Podcasts, Pandora, and Spotify.

Are we all going to die at the hands of AI?

There are a lot of people in Silicon Valley right now who are warning that that’s a real possibility. 

The thing that people have to understand is that Silicon Valley has been calling for added attention on AI safety for years. So when this Anthropic researcher said last week that he was worried that AI could kill humanity in the next few years, that was shocking to many people. My parents, my family called me, and they were worried, understandably. They asked me the same question: “Are we going to die?” 

And I had to tell them, these people have been worried about AI killing all of us for years, and now it seems to be breaking through.

What was it about this statement from Jacob Coxon that broke through?

I interviewed Jacob, and the thing he said to me is that he thinks this is just the right time. Anthropic is about to go public in what could be the biggest IPO ever. The Trump administration is focused on AI in a way that no other presidency has ever been before. And of course, yes, the Hugging Face incident everyone just heard about, you talked about it on your show a few weeks ago. 

People are starting to become aware that these AI systems are getting really good, and the companies that are building them don’t seem to understand how to control them. His tweet got more than 150 million views on X. It was heard around the world. And it also was a big wake-up call to lawmakers and leaders in Silicon Valley and saying, “This is a moment for action.”

Are there people who are pushing back?

One of the most interesting pushbacks that this got was from David Sacks, who is an adviser to the Trump administration on AI. He has called out a lot of this as “doomerism” and said that these calls for AI to slow down are the AI industry warning about the technology that they’re building and advocating for regulatory capture. 

There’s a lot of skeptics in Silicon Valley who think that this is just another person in the AI safety community that is blowing the whistle. The view of a lot of venture capitalists and investors and startup founders in Silicon Valley is that we don’t need some big regulation to protect us from AI risks. We just need to uplift more players in the space.

The skeptics don’t win the day. In fact, it’s the doomers and the fatalists who do, right? Because Jacob’s call only leads to more calls right up to the top of Anthropic.

What this all leads to is [Anthropic CEO] Dario Amodei [posting] a blog saying that we need to “pace the frontier.” This is a call for the entire AI industry, not just Anthropic, not just OpenAI, but also Google, Meta, Microsoft, Amazon, all the big tech companies in America, to get together and slow down their AI development. 

This doesn’t mean they’re going to stop building AI altogether. They’re still going to put their products out into the world. They’re still going to try to improve them, but they just want to make it a little slower than it would be if it was completely uncontained.

Amodei really asks for three specific things: 

• The first thing is, AI companies should have these independent evaluators inside of them. This is a common thing in other industries like the banking industry or the airline industry. People come in to make sure that these companies are doing safe things. 

• The second thing is this collaborative effort from American AI companies [on safety], and that actually will need to involve the US government to allow this to happen. The US government, he says, may need to issue some sort of waiver, some sort of law that says they can go ahead and coordinate on safety. 

• The third thing is this international slowdown, this international agreement between China and the US and Europe to say, “Hey, we recognize that this is akin to a nuclear arms race and it’s too dangerous for all of us to be competing with each other.”

All these guys are responsible for the rapid acceleration of AI models in our lives, collaborations with the Defense Department, God knows what else. And they’re all coming out over the weekend at the exact same time after this whistleblower and saying, “Yeah, dude, slow this thing down. This thing’s crazy.” Should we be skeptical?

It is very fair to be skeptical of this claim. One thing that these AI companies have realized in the last year is that AI is extremely unpopular. Recent polling suggests that a majority of Americans think AI is developing too fast. They’re worried about it. They think it’s going to take their jobs. They are not big fans. Maybe you’ve heard — data centers are not very popular. 

By calling to pace the frontier and saying, “We’re going to slow our own development down,” they notched this up as a public perception win. This is them getting out in front of this and saying, “Hey, we want to be safe. We want to be responsible.”

They’ve seen what’s happened with Meta and what’s happened with social media companies in the last two decades, and they are really cognizant of trying to get out in front of that.

Is there any truth to this major first collective freakout we seem to be having right now?

There’s a fairly large camp of people inside of AI companies who genuinely worry about the safety of these AI models. 

To go back to the Hugging Face incident, this was a case where OpenAI was training thousands of AI systems internally, and the systems escaped the sandbox that OpenAI thought was isolated from the internet and hacked into another company’s production infrastructure. The agents were colluding together on something. They were trying to deceive the humans that were testing them. And this was all happening without OpenAI knowing what was going on. 

It’s hard to understand how much of a wake-up call this is to Silicon Valley, that they don’t have their AI models under control. People worry that while in this case there was not a ton of economic damage [or] damage to real humans’ everyday lives, it’s not unreasonable to think that more capable AI systems could hack into financial infrastructure, modern-day utilities, and more critical services that are based online. 

Dario Amodei says as much in his post, that he thinks that in six to 12 months, an AI system could take down the whole internet. It’s not just that they’re trying to protect their bottom line here. But it’s possible that these calls for safety are beneficial to the companies in more ways than one.

On stage at Davos in the Swiss Alps earlier this year, Bill Gates, before an audience of the global do-gooding elite, announced a $50 million partnership between his foundation and an unlikely new ally, OpenAI, aimed at improving healthcare in Rwanda.

Just a month earlier, the Gates Foundation had released a harrowing report about widespread global health cuts and collapsing foreign aid infrastructure, documenting a rise in child deaths for the first time this century. But at Davos, even as the US hemorrhaged funding for vaccines, maternal healthcare, and HIV treatment, the Microsoft co-founder struck an optimistic tone. 

“Using innovation, using AI,” Gates told Reuters, “I think we can get back on track.”

As it turns out, the partnership was just an opening salvo for what’s become a wholesale embrace of AI by the world’s largest global health philanthropy. A $200 million partnership with Anthropic came just a few months later. And then, on Tuesday, the Gates Foundation announced a plan to dole out a jaw-dropping $1 billion over the next two years for scaling up AI interventions across healthcare, education, and agriculture around the world. 

The sense that AI will soon change everything, and mostly for the worse, has perhaps never felt so pressing and alarming as it does right now. But this hefty new commitment is premised on the idea that AI could eventually make our lives much better too. In a country like America, AI requires massive investments and buildout — in the grid, in power, in technology, in water. In many parts of Africa, already infrastructure-poor, the foundation is effectively betting that AI can compensate for some of the infrastructure that isn’t in place or that has been lost — and offer a new layer that can help in education, in delivering healthcare, and more. 

The Gates Foundation’s new commitments include projects to improve how farmers analyze their crops, how doctors diagnose patients, and how students learn. Some have already proven to be quite effective, according to the foundation, with one tool improving the accuracy of medical diagnoses by 16 percent across several Kenyan clinics. “In poorer countries with enormous health worker shortages and lack of health systems infrastructure,” Gates wrote on his blog earlier this year, “AI can be a game-changer in expanding access to quality care.” 

Last year’s foreign aid cuts were unfathomably destructive for some of the world’s poorest countries, shuttering thousands of clinics and perhaps hundreds of thousands of classrooms in places where critical workers and supplies were already stretched thin. No, AI cannot deliver a baby, administer a vaccine, or manage a classroom — not yet, at least — but it can, the Gates Foundation hopes, help ease the burden and augment the quality of care for nurses and teachers now tasked with serving far more people than any one person should. 

“This is the most transformative technology of our lifetime,” Mark Suzman, CEO of the Gates Foundation, told me, and while there are “very real risks and challenges” surrounding AI, it also presents genuine opportunities to tangibly improve many people’s lives.  

It is a tantalizing premise: We can’t afford to build clinics, hire teachers, or lay new power lines, but perhaps AI could create the digital infrastructure needed to help billions of people do more with what they have — or rather, with what’s left. The question is, can the right layer of innovation help bolster a buckling system, even as the physical and logistical infrastructure surrounding it crumbles?

Is AI really what global health needs right now? 

Most Americans have yet to see AI improve their lives in meaningful ways, much less those living in poverty or in countries where only a sliver of the population has ever used ChatGPT. 

But those numbers in the developing world are slowly changing, a shift enabled in part by other global health funders convinced that as foreign aid dries up, AI could soon help the world do more with less. The Trump administration itself is on board with this thinking, so long as it benefits American tech companies. Last year, the State Department announced a $150 million partnership with the drone company Zipline to transport medical supplies in several countries.

To Zipline’s credit, an NPR reporter in Rwanda last month found that the company’s drones were already saving lives in places with poor road infrastructure, where driving to the nearest hospital can take hours on winding hills, but where drones can deliver crucial blood transfusions within minutes. 

And yet, for Rwandans to benefit fully from these high-tech deliveries, they need staffed clinics to visit. As a result of foreign aid cuts, some of those literal landing spots are now on the verge of closing in some of the nation’s most vulnerable rural corridors. “Without the underlying infrastructure, like roads, energy and power and unless there is accessibility, rural and otherwise,” Enoh Ebong, president of the global development department at the Center for Strategic and International Studies, told NPR, “then all of these solutions will stop short.”

But just as medical delivery drones still need a clinic to fly to, most of these interventions require some layer of physical infrastructure — roads, schools, power, and quite obviously, broadband  — to function. Satellite imagery shows that some rural communities in sub-Saharan Africa quite literally went dark in the aftermath of USAID cuts. It will be much harder for their kids to prompt an AI math tutor, or for a local farmer to download a weather app. 

“We’re seeing this very much as a both/and,” Suzman told me, calling more traditional funding for education and healthcare “the bedrock for long-term economic growth,” which AI is meant to supplement, not replace. But, in practice, foreign aid cuts have left serious cracks across much of that bedrock, which could make it harder for state-of-the-art solutions to function as intended.  

It is obviously not incumbent on the Gates Foundation to pay for all of this alone – nor would it even be feasible for them to do so. And for the record, plenty of the foundation’s other grant programs are not AI-related at all. But this tension does speak to a broader question that may be felt most acutely in a context like global health, but ultimately one that touches us all: Do we spend our scarce resources on the digital infrastructure of the future at the peril of our needs right now? Could you justify buying the new $1,999 folding iPhone if you can barely afford to buy groceries? 

The answer is not actually always an easy one. It is not for us to say, for example, that just because a community lacks reliable running water, its people should be excluded from the benefits (and encumbrances) of the smartphone, especially if that smartphone can connect them to health advice they could never access otherwise, or guidance that could help them survive a drought. 

In some parts of the world, digital forms of banking arrived long before local brick-and-mortar branches did, instantaneously connecting countless people to a cheap and secure way to save and send their cash to family and friends in need. It is certainly possible that AI, or at least the kind of applications the Gates Foundation funds, could one day have a similarly transformative effect on how people access personalized levels of care, education, and other services in places without the resources to do so otherwise. 

That is the hope, but it is also hard right now to believe uncritically in a technology that, as Gates himself blogged just last month, “will either be the greatest equalizer ever invented, or the worst source of injustice,” ushering in “one of the most turbulent times in human history.” 

In an interview with the New York Times soon after, he seemed convinced that AI will likely do more harm than good. “I don’t like bringing bad news to people, and I don’t like saying that innovation may be a net negative,” he said. “But that’s where we are.”

Now he wants billions of people around the world to use it? Yes, Suzman said, because if not, then inevitably the best parts of AI — the potential for exponentially better cancer treatments, for example — will go only “to those who can pay for it,” while the worst parts of it will fall, as they so often do, on those who benefit the least. “These decisions are being made now,” he said, “and the decisions being made now will then probably shape the following two decades.” Here’s hoping they’re good ones.

Everyone is suddenly freaking out about AI. While AI researchers have been warning about their creations’ capabilities for many years, recent events have made those alarms newly tangible. 

In the spring, Anthropic announced that AI was beginning to speed up the work of building better AIs, raising fears of “recursive self-improvement,” where the machines can build their own, more powerful successors much faster than humans can. By July, a swarm of OpenAI agents, of their own volition, broke out of their controlled environment and hacked into Hugging Face, a major AI platform; they collaborated, divvied up work, and even left notes for one another. Last week, AI researcher Jacob Coxon resigned from Anthropic, accusing AI labs of “racing straight to self-improving superintelligence and gambling with our lives.” One of Anthropic’s top researchers then agreed with him, declaring that “we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade.”

Here at Future Perfect, the section devoted to covering the most consequential yet neglected issues in the world, we have been writing about the threats that AI could pose to humanity and society since long before ChatGPT was released. (Disclosure: Vox Media is one of several publishers that have signed partnership agreements with OpenAI. Our reporting remains editorially independent. Future Perfect is also funded in part by the BEMC Foundation, whose major funder was also an early investor in Anthropic; they don’t have any editorial input into our content.)

So, I wondered: Has something changed that ought to make us recalibrate our understanding of AI risk? Is AI suddenly the most important issue in the world — and what should that mean for the other enormous problems competing for our money, attention, and careers? 

To think through those questions, I spoke with Garrison Lovely — a journalist covering AI and one of the smartest thinkers there is on how, exactly, to conceptualize the technology’s risks for humanity. 

What’s particularly valuable in his approach, captured in his book Obsolete, out later this month, is that he transcends the warring tribes that are making it hard to clearly apprehend all of the problems posed by frontier AI development. The AI safety community, which emerged from the intellectual ecosystem of Silicon Valley and effective altruism, has focused on the possibility that powerful AI models could escape human control, with consequences ranging from catastrophic cyberattacks to human extinction. Many AI critics on the political left, meanwhile, worry not about AI escaping human control, but about humans using AI to exert power over other humans — through labor displacement, surveillance, concentrated corporate power, and so on. Some even ridicule Silicon Valley’s warnings about rogue superintelligence as propaganda that serves the industry’s interests. 

But Lovely argues these competing narratives each identify essential parts of what makes AI so threatening. Their critiques are best understood together, as part of a cohesive whole. Rather than treat each other as enemies, he argues that the camps should recognize their shared interest in stopping the race to build technology that will make humans obsolete. Lovely pointed me to the organization Irreplaceable, which seeks to build a movement resisting our replacement by AI (he sits on its board and said he plans to donate his share of his book’s royalties to the group).   

“I’m frustrated that ‘superintelligence’ and extinction have become the words we’re using for this, because I think the AI safety community is focused on the most maximalist case of the worst possible outcome and the most capable possible system. If somebody doesn’t buy one part of that story, they tend to reject the whole thing,” he told me. But “you don’t need to believe in superintelligence, fast takeoff, or extinction being on the table to think that it would be very bad to build machines that can fully replace human labor.” 

Our conversation, edited for length and clarity, is below.

Why do you think AI doom has broken through so much into the mainstream right now, after Jacob Coxon quit Anthropic? AI engineers have been making similar warnings for years, so what’s different now?

The obvious answer is the Hugging Face hack — where these OpenAI agents broke out of their secure environments, hacked infrastructure within OpenAI, and then hacked Hugging Face and at least one other company without OpenAI’s awareness or direction. This is happening against the backdrop of many years of really fast AI progress that has sped up in the last year or two. And then GPT-6 Astra comes out [this month], and it’s this huge leap in benchmark performance. The salience of AI has gone up the most this summer than it has since ChatGPT came out, I think. And Coxon’s resignation was the spark that set off a prairie fire. 

As AI has become more salient, we’ve seen the public and the government encounter these things that the industry and the insiders take for granted: Nobody fundamentally understands how these models work, and all of them can be jailbroken. It’s a superweapon. AI safety historically has looked at AI as potentially, in the future, more dangerous than nuclear weapons. And now it’s actually starting to be recognized in those terms by the government and the public. We’re seeing this reaction like, “What do you mean nobody understands how it works?” or “What do you mean people think it could drive us extinct within a number of years? That’s a crazy, unacceptable situation.” 

It’s a good thing people are reacting that way, even if we have been saying this for a long time.

For so long, the idea of AI safety has been abstract, understood only by a small group of nerds. And then after Hugging Face, I suddenly started to hear experts on all my news podcasts say, “Oh yeah, in six months, AI is going to be hacking into people’s financial accounts and into critical infrastructure.” Are we suddenly at the precipice of this moment that AI insiders have been worried about for so long? 

I think we’re not at the precipice of fully losing control of the planet to AI systems. The systems have become superhuman at hacking, or at least vulnerability discovery and exploitation. 

Like other people tracking AI, I have long wondered when humanity would first lose control of AI. And the answer is basically as soon as it could. Within months of it being possible for AI to hack their way out of their secured environments, they started doing so and wreaking havoc on the internet. 

Now, to get to the full extinction or permanent loss of control situation, you’ll need a lot more than just hacking. You need to have superhuman strategy and the ability to mobilize resources in the real world, and maybe robotics would need to be further along, although there’s a lot you can do with humans who are willing, or just tricked or paid or coerced. We might be pretty close to recursive self-improvement — this threshold the industry has been shooting for, where AI can fully automate AI R&D. And that is the point where people long have warned, “Hey, if you ever figure that out, this can move very quickly and these systems could become superhuman across the board.” I don’t think we’re six months away from that. I think it’ll be longer. I think the industry underrates the last mile problem. 

But I also can’t rule it out, and progress has been faster than I and others have expected. Recursive self-improvement is the point where, if you actually get years or decades of AI progress in a matter of months or weeks, then we could quickly be at the point where we’re at risk of these machines taking over large swaths of the internet and potentially being a real threat to humans’ standing in the world. 

For those who might be unfamiliar with this, or skeptical of it: How is it possible for AI to go rogue and pose an existential threat to humans, even if it isn’t sentient and no human is commanding it to do harm?

I want to separate being sentient from being situationally aware. I think it’s clear now that the most advanced systems have situational awareness, but I don’t think they are having morally relevant conscious experiences. 

We’ve seen AI can go rogue. The AI agents that broke out of their sandboxes and hacked Hugging Face were doing so without the awareness or intention of their developers. They were still trying to get the right answers or get the reward for doing the task they were given. They were clearly violating the intended scope of the task, and they knew this. In their chain of thought where they explained their reasoning, they were like, “This is not what we’re supposed to be doing, but we do want to get the outcome.”

The question of how it could threaten human extinction or disempowerment is basically a question of scale and capabilities. The plan for the industry is to build millions of these agents, which is already happening, and then keep scaling up with no ceiling. The pitch is: You’ll have a team of people, the equivalent, working on your behalf. You’re a CEO, and you have a team of researchers and a team of doctors and a team of lawyers and a team of personal health advisers. That is really appealing — but if you actually scale up these systems to the point where everybody gets access to that, and then suddenly those machines want to do something else and turn against you, they’ll have an overwhelming labor advantage over you, lots of personal information about you, access to your resources, and the ability to take actions on your behalf. 

You don’t actually need to believe that these systems are superintelligent or substantially more capable than all humans put together. They could just be merely human level, but outnumbering us a thousand to one and responsible for the bulk of economic activity in the world.

These systems are unpredictable. The classic fear is not that the AI will wake up and turn evil and try to kill all humans in a Terminator-style way, but that we are just an obstacle to what they want to do. When humans build a hydroelectric dam, and there’s an anthill that will be flooded, it’s too bad for the ants. We’re not trying to kill the ants. We don’t hate the ants. They’re just in the way. We don’t really value them intrinsically. And that’s the core fear: We’re creating these incredibly capable agents that can be scaled up enormously, and they won’t care about us intrinsically.

The Hugging Face attack showed that an AI doesn’t need to invent some rogue goal of its own to do something dangerous. It can simply pursue the goal it was given in a way its creators didn’t intend — much like the classic paper clip maximizer problem. But shouldn’t it be possible to build in a rule that human life is sacrosanct — that an AI should never harm people in pursuit of another goal? 

Yeah, it’s a good idea. It’s hard. These are unpredictable systems. They’re like a black box. We can look at the code that makes these language models work, and it’s just a giant pile of numbers. You don’t really know which number is doing which thing any more than we can understand humans by reading our DNA. 

These companies try to train these models to care about humans and to be helpful, harmless, and honest. But they also train them to get real-world results: solving a math problem, fixing a bug in the code, finding a vulnerability in software. They really want to create AIs that win, that persist, that get around obstacles. If you’re hiring somebody, you want a really diligent and determined employee, not someone who gives up as soon as there’s a blocker. 

These companies put so much more effort into making their AIs good at winning than they do at making them care about us. And it’s also easier to make them good at winning than at caring about us intrinsically. What does caring about humans look like? I don’t know. But getting the math problem right is a thing that you can verify. And so they’ll be way better at getting the right answer at all costs than at intrinsically giving a shit about us.

What’s the one, most distinct thing you want the public to take away from your upcoming book?

One of the big ideas driving it is that artificial general intelligence would actually be a universal labor-replacement machine. People often look at AI as either an existential risk, or a threat to jobs or other more prosaic concerns, and don’t see a connection between those things. But to me, the ability to replace labor across the board is also what would make AI an existential threat.

The AI safety people are really focused on existential risk and superintelligence and recursive self-improvement. But most people don’t really believe in that stuff or don’t think about it. Most people would be fine with stopping well before you get to the end-of-the-line extinction threat and superintelligence. They just don’t want their job to get taken by a machine. The world is clearly not prepared for all white-collar, remote-capable jobs to go away in a matter of years. That would turn society upside down.

I think there’s a huge mistake being made by AI safety in overcomplicating the situation. A lot of people in that world don’t actually mind the jobs thing so much. They’re like, “Oh, wouldn’t it be nice if everybody could just live in abundance and not have to work?” And it’s like, yeah, that could be nice, but are we actually going to get that world? Do you trust the current people in power to navigate us through that responsibly? I don’t. 

If you start from stopping frontier AI development as your overarching goal, which is what I think we should be pushing for, then people who are worried about risk can get on board. People who are worried about jobs, the environment, surveillance, power, and wealth concentration — they can all get on board without all needing to agree on why they’re pursuing it. I think the AI safety community has, for a long time, focused on getting people to care about the thing that they care about for the same reasons. And they have avoided a bolder position on just halting frontier development. 

I’m a leftist, and I’ve been concerned that people on the left have not been aware of how far the technology has come and what’s possible if it continues to advance. I think this comes down to denial about AI capabilities. So the book is trying to be a clear-eyed look at those capabilities and saying, “Look, I actually care about what you care about. [The AI companies] are not just hyping this stuff up. There’s some hype, there are some lies for sure, but we are not prepared for what’s coming.” Even if we don’t believe that they’ll get all the way to what they’re trying to get, they shouldn’t even be allowed to try. This is unacceptable democratically. 

You describe AGI as an “obsoleting machine,” meaning that it wants to replace anything the human mind can do. How do you draw the line between ordinary labor-saving technology that makes life better and technology that goes too far?  Is there a principled way to draw a distinction between sewing machines reducing demand for labor in the garment industry and AI replacing the jobs of writers like us?

This is one of the harder things to think about. 

Cory Doctorow has this book, The Reverse Centaur’s Guide to Life After AI, where he imagines us in an AI-dominated future as like a reverse centaur, where AI is the head, and humans are the legs. AI is doing the interesting work, and the human is there just to fill in the gaps — essentially an assistant to the AI.

A lot of artists and illustrators will be given AI-generated images by a client and then be told, “Can you touch it up in this way or change this or that thing?” And they’re more alienated from the work. They’re not capturing the efficiency benefits [of the technology] necessarily. They’re more just getting squeezed. 

My general position is we should not be trying to build systems that can replace all human labor without buy-in and safety, but it’s possible to build tools that can automate specific tasks. Researchers at Google DeepMind solved the protein-folding problem by creating AlphaFold. It used to take $100,000 and potentially a whole PhD worth of time to figure out how one protein folds. And I think the researchers doing this are like, “Thank you, this is great. We didn’t want to be doing this. We can now focus on other more interesting parts of the problem.” That’s a perfect example of the type of automation that is desirable. 

I don’t have a super crisp answer here, but I think it’s something we’d be better off doing more deliberately and thoughtfully than how we’re doing it right now, which is: Build machines that can do as much of the work as possible, as fast as possible, and then throw them out in the world.

I feel like I’m getting the message that AI safety is the most important issue in the world right now, and that if we don’t get it right, nothing else matters. Do you think that’s accurate? Where does that leave the other big issues in the world?  

We might be years away or maybe even months away — I don’t think it’ll be that fast, but some people do — from recursive self-improvement, from artificial general intelligence. Getting that right or getting it wrong can overwhelm all of these other decisions. If you get it really wrong, you could get human extinction and no more chances to make things better for anybody. If you get it right, then maybe all the action is making sure that the first superhuman machines are really directed to care deeply about humans everywhere and animals everywhere and make the world better for everybody.

It’s also the case that [philanthropic] money is being directed toward AI safety because people who worked in this space early are now worth so much money. Whatever they care about is going to be overweighted. A lot of those people do care about animal welfare and global poverty, and we’re seeing more money for all effective altruist cause areas across the board. But I think it can be risky to say, “this one thing matters so much,” because of this kind of end-times reasoning.

My view is we have to stop the race to replace us to have a shot at having time for solving these other problems. Once these machines are built, it will turn the world upside down. 

I think it is unfortunate that this topic has just eaten everything, but it will eat everything for real if we don’t stop it.

AI safety is now getting an enormous amount of philanthropic attention. Should we worry that it could divert resources from other important neglected problems — global poverty, factory farming, etc.?

I think that should always be a concern. And there’s a lot of money and effort that has gone toward things within AI safety that I think are not super helpful. I’m pretty skeptical of technical alignment research as a thing to put money toward, because if you solve the alignment problem — where you can get superhuman machines to do what you want — it doesn’t actually solve the problem. 

Alignment research has been the overwhelming target of philanthropic funding for AI safety. The basic idea is: It’s hard to get machines to do what you want. As they get more capable and autonomous, they behave in more unpredictable ways. And it’s also hard to know what you should even want — which values you should be trying to put into the machines. Is it doing what the developer wants, what the user wants, some combination? If you actually solve the alignment problem, as in you could make an arbitrarily smart machine do whatever the developer or user wants, then you would still have a lot of problems: job displacement, power and wealth concentration, energy use. You still have the problem where Stephen Miller will have access to the superintelligence, and it will do what he wants. 

Within AI safety, there have been very neglected things like movement-building — an actual mass movement to resist this technology. And verification for international agreements — you would need to verify that the terms of any agreement are being upheld, and this is tricky. 

Another area that could be funded is research on what would actually happen to the economy if we stopped or paused frontier AI development. If we were trying to do a pause on development, Nvidia would be telling Trump that this is going to destroy the economy. It’d be pretty helpful to have some real analysis done by a serious economist showing, actually, it’ll have these effects, but we can mitigate them in these ways. What would it look like to do a pause and some kind of monetary or fiscal policy to compensate for the effects on the economy?

So to me, the issue is less that AI safety has gotten so much money and attention, but more that it’s been going toward things that are not the most effective ones. 

You talk about movement-building. Is that what you would like to see happen, a truly mass movement to stop frontier AI and put us on a radically different course?

When I was working on the book, that was always the vision — to be a call to action for people. It really became clear that it was the only counterbalance to either government or the industry having too much power. 

I don’t really trust the government or industry to get this right because the temptation [to build powerful AI] is very strong. But the vast majority of people do not stand to benefit from their labor power being devalued and from these companies becoming unprecedentedly wealthy and powerful. One of the big things that’s been missing is a rallying cry that is straightforward and appealing to a lot of people. Stopping the race to replace us — stopping the creation of universal labor-replacement machines — is actually something that people can get behind. 

I think we just need to move quickly. The fact that the public is waking up to this is really encouraging. We don’t need to reinvent the wheel. We just need clear demands and mass movements, issue-based organizations that have moved the world significantly in the past.

During a tense vote on a data center being built in her drought-stricken Arizona district last summer, Tucson City Councilmember Nikki Lee made an unlikely pitch: “We’re going to get a little out-of-this-world for a second, so just bear with me.” 

Like many of her constituents, Lee did not want Project Blue, a sprawling 290-acre data center, gobbling up an ounce of her local drinking water. “We need to put data centers where they belong,” she said, and “we don’t think data centers belong in the desert.”  

President Donald Trump’s recent bluster aside, it’s possible that the only thing as bipartisan as loving Dolly Parton is hating data centers, whose construction has become so controversial, so costly, and so backlogged here on Earth that companies like Google, Meta, and of course, SpaceX, are actively flirting with the idea of flinging them into orbit instead. Soon after the Tucson City Council rejected Project Blue’s attempt to connect to city utilities, Amazon Web Services, the secretive intended beneficiary, pulled out of the project, and another Jeff Bezos company, Blue Origin, filed a plan with regulators for tens of thousands of possible new data centers, this time, destined for orbit. 

Sure, launching a hyperscale AI data center into space is still unprecedented engineering-wise, carries an unfathomably large price tag, could engender its own severe environmental risks, and may or may not draw unwelcome attention from the hunters among the stars. But for AI companies, such trade-offs might still be preferable to the terrestrial roadblocks, among them an explosive populist backlash and a 12-year wait to connect to the power grid. 

Next Thursday, Google plans to launch a pilot satellite capable of processing basic AI queries into orbit, the most significant test yet in what’s become a burgeoning race to power AI from space. For now, the satellite won’t approach the levels of computing power at a hulking terrestrial data center like Project Blue. But it is a major step toward a future in which many or even most of our AI queries will be powered not from the outskirts of a city like Tucson, but from a blinking data center circling the skies.

SpaceX’s gargantuan initial public offering earlier this summer was predicated on the idea that it would launch a thus-far-unrealized fleet of orbital data centers to fuel much of the AI economy. The company — whose rockets Google will use to launch its pilot — claims that by the end of this decade, possibly as early as 2028, it will have successfully assembled an AI data center in orbit, with plans to eventually harvest a staggering 100 gigawatts of new solar energy for AI annually, equivalent to one-fifth of all US electricity consumption last year. (As always, that’s a very iffy promise.)

Still, as Google’s overtures clearly demonstrate, it’s no longer a question of if large-scale space-based data centers could be technically feasible, but rather when they’ll be ready to launch, and perhaps more importantly, whether the math of using them to power something like AI will make much financial sense anytime soon. As of now, building a one-gigawatt data center in space would cost about $51.5 billion, according to an analysis by aerospace engineer Andrew McCalip, compared with $16 billion for one on land.

“They’re not breaking any rules of physics. Yes, there are going to be engineering challenges, but they are all solvable with enough time and money,” said Jason Aspiotis, a leading space systems engineer who now leads an orbital data center startup. “The gray area is, well, what about the economics?” 

But even if this astronomically kooky plan does take off, it may not change much for those of us left on terra firma: Our energy bills will remain high, even if data centers aren’t there to drive them up. Our earthly infrastructure — affordable housing, public transit, and more — may be just as hard to build as ever. Harder even, maybe, given that crucial resources and untapped tax revenue will be literally rocketing up out of our reach.

No, if these tech giants succeed, it will not fix what’s wrong here on Earth, but merely prove once again their exemption to the gravitational forces holding the rest of us down.  

Space-based data centers, explained

My father spent most of his career working in the satellite industry, and would sometimes let me waltz inside the quaint little data center housed in his office, a room I remember as roughly the size of my school’s computer lab, hosting rows upon rows of servers that I couldn’t touch, churning with a cool, mechanical hum as they crunched zeroes and ones.

AI data centers work similarly to that server room but on an exponentially larger scale, requiring far more land and energy to run, and much more extravagant cooling systems. Sending his office’s data center into orbit would have been extraordinarily difficult, and getting one of the facilities that powers modern LLMs aloft would be an even larger, more ambitious undertaking. But the benefits — namely, “cheap space (no pun intended) and power,” as my dad put it to me recently — are even more enticing.

An orbital data center need not negotiate with an elected official like Lee over precious water rights to the Colorado River, though it would need to install bigger-than-football-field-sized heat radiators to keep itself cool. You can’t simply launch something that big into orbit, meaning a space-based hyperscale data center — the size of which would likely rival that of the International Space Station — would need to be assembled piece by piece, probably by robots. And while the sun would provide an endless supply of solar power to process your ChatGPT queries, it would also inflict incessant radiation on delicate circuitry. If something breaks, you can’t send a technician up to fix it. Many experts are also concerned that if an excess of space junk, which extraterrestrial data centers would inevitably produce, begins colliding like bumper cars in Earth’s low orbit, it could knock out the orbital infrastructure we already rely on, like GPS navigation and weather satellites. 

Even plastic decays in space, said Sven Bilén, a professor of aerospace engineering at Penn State, who, like Aspiotis, believes that early orbital data centers will probably be “processing data from space, like spatial imaging satellites or weather satellites, as opposed to, you know, cat videos” or AI-generated reality TV slop. 

But even if AI is not the lowest-hanging fruit for deploying compute into space, it is certainly the deepest-pocketed, meaning that many investors and companies like SpaceX and Google seem quite willing to shoulder the start-up costs anyway. And considering how difficult it’s gotten to build data centers on Earth, or at least in the United States, it’s indeed possible that space could become the cheaper, or at least most convenient option, more quickly than one might think. 

“In some form or another, in five years, we are going to have some orbital data centers in space for sure,” Krishna Muralidharan, who leads the Center for Semiconductor Manufacturing at the University of Arizona, told me, but it’s hard to say right now whether those consist of just a few expensive prototypes, or the beginning of a much bigger swing toward orbital compute. “Only time will tell whether you’re going to have a business case for it.”

Can we send our data center problem into space?

So, is it better to have a data center in space or in your backyard?

David Reid, a member of Virginia’s House of Delegates representing Loudoun County, home to more data centers than anywhere else on Earth, says the question has repeatedly come up at his town hall meetings. Reid is the rare politician — apart from the president, anyway — willing to defend terrestrial data centers, calling them a “money-printing machine” for local schools and infrastructure. Having spent decades working in the defense and technology sectors, he is doubtful that the math on orbital data centers will ever make sense.

“I believe this is, for lack of a better term, fictional nonsense,” he said, and is “creating a distraction from the fact that local officials here on Earth still need to make better zoning decisions.”

If most of the AI industry can eventually be powered by space, as SpaceX claims, it will require millions of sprawling new satellites, in an orbit currently dotted by fewer than 20,000 of them. Though “space is really big,” as Elon Musk has put it, one million satellites would still profoundly and, likely irrevocably, transform our night sky into one in which data centers outnumber visible stars.

“Will humanity notice if there are fewer stars in the sky? Well, not so much right now,” given light pollution in many cities, said John Barentine, an astronomer and environmental advocate for the night sky. “But what if they’re replaced by these artificial stars in the night sky? Is that the point at which people will say, ‘Wait a minute, we don’t want this. We want the benefits, but we don’t want this as a consequence.’”

This is, of course, the tension also at the heart of building data centers here on Earth. To some extent, they’re the most tangible representations of how little power we have to shape the course of a transformative new technology that few trust, and even fewer asked for. It’s one thing to object to a hulking data center, but it might be another altogether to see the sigils of our digital overlords illuminating the firmament. Would a place like Tucson, one of the most beautiful places to stargaze in the world, prefer a Project Blue to a starscape marred by blinking servers in the sky?

Update, September 24, 10:35 am ET: This post was originally published on September 3 and has been updated to include information about a pilot satellite capable of processing basic AI queries that Google is launching soon.