We’re living through an AI-fueled “Black October,” and things are changing fast. Hackers are using AI to breach countless companies. AI tools have uncovered security flaws that could leave 70 percent of all websites vulnerable. Governments are scrambling to secure their sites. Banks, too. And because cyberhacks can hurt supply chains, some longtime AI watchers are going full prepper, stocking up on food, cash, and more.
In the center of this story is Anthropic, one of the world’s leading AI companies, which was explicitly founded on the promise that it would be a “good guy” company in a world of dubious tech giants.
Anthropic says all the latest developments are very serious. Its security chief wrote last week that things are “going to get a lot worse before they get better.” Two weeks ago the company warned that AI models could soon self-replicate on the internet-connected devices in your home, like smart fridges and robot vacuums. And its co-founder now says “the singularity” — when AI systems speed up technical progress in a way that’s hard for humans to comprehend — “is here.”
You might think, given its view of the risks, that Anthropic would be pumping the brakes on its cutting-edge AI progress. Just the opposite. Anthropic filed for a blockbuster initial public offering on Monday, chasing a valuation of $2 trillion even as it warns in the IPO filing that AI could “pose catastrophic or existential risks to humanity.” Every day, it is racing toward recursive self-improvement (RSI), where AI models are smart enough to build the next, even smarter AI models by themselves.
How does Anthropic justify that?
As its CEO, Dario Amodei, reminded the world last week, he built the company on one big bet: that instead of driving a race to the bottom, where companies skimp on safety in their rush to beat competitors in the market, Anthropic could actually catalyze a “race to the top.”
As I’ve watched Anthropic double down on its “race to the top” strategy, I’ve been uncomfortably reminded of another very risky gambit: Sam Bankman-Fried’s crypto fraud of 2022. There are important differences between the two; this is not a perfect parallel. But both involve a tiny handful of figures, strongly influenced by effective altruism, making huge bets on behalf of many, many other people.
EA culture prides itself on offering a counter to commonsense morality. The community has shown a willingness to take big risks and justify them with complex calculations. Sometimes that pays off. But believing you are intelligent enough to adopt a counter-intuitive, galaxy-brained strategy and end up saving the world with it often involves some hubris. And hubris makes me nervous in a field as high-stakes as AI.
The idea is that if Anthropic proves it can build AI that’s both powerful and safe, the market will expect both — so other AI companies will then feel compelled to compete on safety, too. That may sound nice in theory, but it means Anthropic has to keep building ever-more-powerful AIs.
“The forces that drive the race to the top are hugely more effective if the company practicing them is at the frontier,” Amodei wrote. “And looking at that strategy as a whole, I’m proud of our track record and stand by it.”
In other words, Amodei is sticking to his bet.
But Amodei is gambling with, by his and his company’s own telling, the future of humanity. If AI goes wrong, it could lead to our gradual disempowerment, a galling concentration of power, or even complete extinction.
So there’s a very basic question that the public deserves a good answer to right at this pivotal moment: What is the evidence that Anthropic’s stated strategy — compete at the frontier so as to drive a “race to the top” — is actually successful overall?
From the outside, it looks like Anthropic has done more, on balance, to drive a race to the bottom. (Disclosure: Future Perfect is funded in part by the BEMC Foundation, whose major funder was also an early investor in Anthropic; they don’t have any editorial input into our content.)
In fact, part of the problem may be that Anthropic continues to treat this as a classic arms race. Specifically, the company seems to model it as a prisoner’s dilemma: If Anthropic stopped building more powerful AI, another player — whether it’s China, OpenAI, or another company — would just build it anyway. In this kind of dilemma, if the other player is going to defect, it’s rational for you to defect, even though nobody actually wants that outcome.
But what if this framing misses the point?
Game theory can make a huge gamble seem sophisticated, while making the obvious alternative — just don’t build a doom machine — seem naive. Yet the fact that Anthropic and others are now racing toward RSI makes it much more likely that we could lose control of AI and start to see massive damage to our world. When the cost of defecting from safety is that you create a serious risk of catastrophic harm — or of all humanity, including you, going extinct — is that still a “rational” move?
Instead of telling yourself that you can galaxy-brain your way through this with fancy enough game theory, maybe you’re better off just thinking of this as something much simpler: Russian roulette.
More and more leading voices in AI are starting to recognize that.
“We should stop AI progress now, starting unilaterally with Anthropic and/or OpenAI stopping — and then broadening out as a second step to more AI companies and negotiations with China,” Geoffrey Irving, who was until recently the chief scientist at the UK AI Security Institute, told me. “Unilateral pauses can’t hold for long, but they are the right first step: When you’re doing a very dangerous thing, the correct action is stop, even if others are also doing dangerous things.”
Anthropic did not respond to a request for comment.
Let’s be clear: The question is not whether Anthropic has done anything good on the safety front. It has. Standing up to the Pentagon, temporarily holding back access to frontier models, and publicly advocating for safer AI policies are a few examples, among others.
But the real question is whether the company’s “race to the top” strategy is working on balance — whether it’s net-negative or net-positive for safety.
Nathan Calvin, the chief strategy officer at AI safety advocacy organization Encode, recently set off a firestorm among tech watchers, writing on X, “When I look at many notable actions that Anthropic has taken over the past year, that evidence seems most consistent to me with the view that Anthropic is primarily trying to win the race to superintelligence, and that other considerations (including safety) are fundamentally secondary.” Here’s his list of what Anthropic has done:
And this is all just in the past year. If we rewind the tape further back, we see Anthropic pushing back on a landmark bill to regulate AI. Rewind even further back, and we see indications that Anthropic’s core strategy may have been flawed from the start. According to Reuters, in late 2022, OpenAI CEO Sam Altman heard rumors that Anthropic was building an AI-powered chatbot. So he ordered his employees to ship an OpenAI chatbot first. That chatbot was called ChatGPT. We all know the rest.
The rivalry between OpenAI and Anthropic seems not only to have hastened the arrival of generative AI, but also to keep pushing release dates forward to this day. The two companies have been known to release new or cheaper models within hours of each other, each one keen to protect its market share.
Even if you think Anthropic’s behavior was justified in 2022 because publicly demonstrating the risks of AI also demonstrates the need to regulate AI — a common claim within the company and AI discourse writ large — you have to ask: Is it still justifiable now, when AI has reached this level of power and sophistication?
Anthropic has indicated that it’s getting closer and closer to achieving RSI. In January, Amodei himself wrote, “I am concerned that LLMs are approaching (or may already have reached) the knowledge needed to create and release [biological weapons] end-to-end, and that their potential for destruction is very high. Some biological agents could cause millions of deaths.” And he wrote in September, “Given the accelerating rate of AI capability development, it’s my worry that in 6-12 months such a swarm could be capable of taking over the entire internet with a persistent botnet.”
But at this point, Amodei and his staffers are themselves perhaps the primary ones accelerating the rate of AI capability development.
“Anthropic seems to be doing the whole RSI thing more than anyone else, other than perhaps OpenAI,” Calvin told me. “And I just think that seems insane given their publicly stated views on the risks.”
“I think people at Anthropic should reckon more with the fact that it seems like the modal outcome is that the largest impact that Anthropic will have had is dramatically accelerating the race to RSI,” he continued. “On the current trajectory, everything else they will have done and thought and ideals they had will be a rounding error compared to their contribution there.”
Anthropic should be willing to invite scrutiny on this point — and, if it believes it’s actually making the field safer than it otherwise would be, to show us the receipts. How? It could put before Congress a bill that actually strengthens safety, instead of weakening it. It could stick to its former Responsible Scaling Policy and get other companies to do the same, instead of jettisoning its commitments. It could influence other labs not to build RSI, instead of leading the charge.
In the absence of such evidence, the “race to the top” logic just doesn’t hold water.
Amodei has begun advocating for “pacing the frontier” of AI development. “We must slow the pace at which we improve the capabilities of AI models,” he wrote in September.
The catch is that Amodei’s plan to slow down is contingent on others agreeing to slow down too — other AI labs, to start with, and also other countries. But more experts are now saying that it would be better if a frontier company like Anthropic unilaterally stopped.
“I’m glad that he proposed that we pace, but like, now we have to actually do it — and it seems like he’s saying he’s not going to do it unilaterally,” Daniel Kokotajlo, a former researcher at OpenAI who became a prominent whistleblower, told me. “That’s not very helpful, Dario! You should lead. What’s all this about ‘racing to the top’ and ‘leading by example’ if you’re just waiting?”
Voluntary unilateral action by individual companies is of course no substitute for federal regulation and international treaties. But Irving argues that it’s the right first step, because if the companies wait for governments to rein them in, it may well be too late. Governments are notoriously slow. An individual company can act today.
There’s a perennial objection here: “But China!”
Yet the idea that we’re in a classic arms race with China has always been flawed.
For starters, AI is not just one thing with one purpose, like the atomic bomb. It’s a much more general-purpose technology, like electricity. So what matters isn’t simply who gets it first, but who uses it in which ways. Besides, if the harms from AI are massive, then those harms are relevant for the other party, too. The Chinese don’t want to see catastrophic harms to all of human civilization, either. So we shouldn’t write off the possibility of coordination.
Former Vox reporter Kelsey Piper recently wrote about a colloquium at Anthropic where one participant objected: A decade ago, Facebook considered acquiring the tech that would later power TikTok. But, facing pressure due to privacy and safety worries, the company passed. That didn’t end up protecting us from the scourges of a world with TikTok — all it did was hand China an early monopoly on the tech. Piper writes:
Was this a policy success? Did we want to repeat it?
We were obliged to grudgingly concede that it didn’t sound like a policy success. We did not want to repeat it. China might be no excuse for the U.S. labs to do reckless things, but it is a real consideration.
Our next step might be bilateral talks with China, then. But by the point where you agree that Anthropic’s plans might depend on bilateral talks with China, you’ve already conceded most of the steps of Anthropic’s worldview that seemed so much like madness 10 minutes ago: continue working on AI for now, try to negotiate a slowdown, but don’t abandon work at the frontier and all the advantages that come with it.
But TikTok is not a good analogy for RSI. The former may be a cesspool that rots our brains and amplifies ethnic hatred, but its dangers don’t compare to the dangers of RSI. The TikTok case may indeed be best approached as a classic prisoner’s dilemma. But with RSI, the potential downside is so catastrophic that anyone who proposes to save the world precisely by building the tech that endangers it is making a hell of a gamble on behalf of everyone else. At the very least, the burden of proof should be on them to prove their confidence is justified.
AI insiders often think those outside of their industry can’t possibly understand the technical, geopolitical, and moral complexities they’re facing. But commonsense morality is commonsense for a reason: It tracks intuitions that are usually right (such as, if you don’t want a doom machine, maybe don’t build a doom machine) and trains us to be appropriately skeptical of huge, counterintuitive bets to the contrary.
Precisely because the AI landscape is a complex system — there are so many actors (including different AI companies and different governments), and actions from each can trigger actions from others in ways that are hard to predict — believing you’re smart enough to take highly counterintuitive actions and end up saving the world as a result is an extremely risky gambit. Instead, maybe it’s best to just do the straightforwardly right thing: Stop.
To be clear, “stopping” doesn’t have to mean peeling back everything that’s been done in AI so far. The powerful AI systems that are already out there can’t be put back in the box. And we may need some AI to defend from the AI that’s already out there.
“I’m not saying shut down all of your data centers,” Calvin said. “But I’m saying that in particular, pushing the frontier on RSI and making it so that you’re going to be the company that is doing more fully autonomous development than anyone else in the world, does that actually make any fucking sense?”
Increasingly, both AI insiders and the public are shouting no.
Some insiders, perhaps aware that humanity is perfectly capable of shutting down overly dangerous tech development (we’ve done it before), are speaking out and quitting jobs at AI companies. At public gatherings, like the recent Pro-Human Assembly in Washington, DC, and the People’s AI Assembly in New York City, people are piling on the pressure to rethink the AI race. States are drawing up their own legislation to rein in the tech. And a single X post from Calvin got so much attention that no less than Amodei himself felt compelled to respond. The AI companies are being forced to pay attention.
Attention may not be all you need, but it’s a start.
There are some days when there’s so much that needs to get done, I daydream about having a clone of myself. Between paying bills, life admin like renewing my driver’s license, keeping up with doctor’s appointments, and planning meals, it’s a lot. And that doesn’t even begin to touch the work that goes into making a podcast or managing my social life.
This feeling of overwhelm resonates with Sari Azout. She’s a mom of three who works in tech. “There is just so much doing,” she told Explain It to Me, Vox’s weekly call-in podcast. “I wake up, and I have to do meal coordination and get my kids up for breakfast, coordinate carpools. Then a lot of work during the day, but there’s the email from the teacher, there’s the 20 group chats asking if you want to participate in this group gift. They don’t call it ‘the invisible load’ for no reason.’”
Silicon Valley says it has a fix for this kind of mental overload. They don’t want to clone us — at least not yet. But they do think they have the next best thing: AI agents.
Azuot was skeptical when she first got an invite to try an agent, but she says she eventually came around on the technology after back-to-school shopping snuck up on her. She used her agent to get it done. “It blew me away,” she said. “I remember that night coming home just feeling so calm and reading my son a book and feeling present. And I was like, ‘Wow, I actually was very cynical about something that is giving me more time for living.’”
Azout’s experience is, more or less, the sales pitch for AI agents: They’ll help you save time with everyday tasks, so you have more of it for what you want to do. But what are agents, exactly? Are they really all that? And what do we gain — and give up — when we use the technology?
On this week’s episode of Explain It to Me, I spoke with Reece Rogers, a senior software reporter at Wired, to find out. Rogers says to think of agents as the next evolution of AI chatbots like Claude and ChatGPT.
“I’d create this mental model of a tiny digital assistant that’s constantly searching the web, going through your email, and checking your calendar, trying to find ways it could proactively be helpful for you,” he explains. “Agents are more proactive. It’s scanning your email 24/7 and sending you a message, maybe highlighting a potential calendar conflict.”
Below is an excerpt of my conversation with Rogers, edited for length and clarity. You can listen to the full episode on Apple Podcasts, Spotify, or wherever you get your podcasts. If you’d like to submit a question, send an email to askvox@vox.com or call 1-800-618-8545.
What agents are out there right now?
At the start of this year, there was a tool called OpenClaw that really sparked this with early adopters in San Francisco, but now there are much more consumer-friendly versions, with one of the most notable being Meta’s Muse.
There’s also a tool from OpenAI called Dots. But where Meta’s Muse is free to download, OpenAI’s Dots is still behind a $100/month paywall.
You tried an AI agent for a recent story you wrote. How was that experience?
One personal example, to share something that I’ve actually been using agents for, is our couch. It’s broken, and finding a new couch has been a pain. We want a sleeper bed, but we want these specific colors. We have this specific budget. I’ve been using Meta’s Muse and OpenAI’s Dots to try to find a few different options for us.
One interesting thing is that Meta’s Muse is connected to Facebook Marketplace. There are these really good integrations where I can give it all these details, and it found a few good couches that are only maybe a couple blocks from me.
I really enjoyed OpenAI’s Dots experience as well. I spent about an hour talking to it on the phone, and the AI agent made a packet of all these different options I could find. Just this morning, it flagged that one of them is now 40 percent off.
Is that how this tech is typically used?
Most people are using this as part of a workflow. I attach it to my email, I attach it to my calendar, and I’m using it as a personal assistant to organize my life. But people have been quite creative.
One of the breakout uses for Meta’s Muse is people have been connecting their bank account statements and then using it to scan for subscriptions they might want to cancel. So it can be used in many different ways, though I will say I do not connect it to my own bank account for a myriad of reasons.
I’m not going to lie, the idea of giving any third party access to my bank information does not thrill me. I don’t love that concept.
That’s one of my biggest concerns about these AI agents: how much personal information we’re handing over to these large companies.
When I tested Meta’s Muse, it wanted access to my email, my bank account, even my passport expiration details. The pitch is convenience, but Muse users are automatically opted in to having all of their interactions with Muse used to train Meta’s AI models.
I would definitely recommend, if you’re sharing personal details, go into the settings and turn that off because you might be handing over even more information about yourself than you initially realize.
What is the privacy situation with all of this? We all have various apps where we don’t read and still click “I agree,” but this feels like it could be more invasive.
Yes, especially because it’s all about automation. If you’re automating things and it has access to all of your data, it could make mistakes and embarrass you in ways you might not even be thinking about.
There was this one really interesting story of someone who had connected an AI agent to their work Slack in addition to their bank account. And their agent just automatically posted how much money was in their checking account for all of their coworkers to see.
Oh no.
I would be mortified. So there are these kinds of mistakes that these AI tools can still make. If you’re interested in experimenting with them, I would be really thoughtful with how you’re setting up these tools as well as what kind of information you feel comfortable sharing.
“You can see in the marketing that these companies are trying to find some way to entice new users.”
How are these AI agents being marketed? AI is not popular right now. Just ask any politician who has a data center coming to their state. People are not loving it. How are they trying to sell this to the average person?
The AI backlash really is at a fever pitch right now. You could almost see that in the way that AI companies are trying to market these tools. They’ve really, really leaned into cuteness and into adorable mascots.
Muse, if you haven’t seen its agent, looks like a mix between a Labubu and an Ewok. It’s very furry, very cuddly, very cute. OpenAI’s tools also have similar kinds of blobs with large eyes that might be a little endearing.
I feel like companies are hearing this kind of feedback that people aren’t trusting AI right now, or maybe they have moral or ethical reasons they don’t want to use these tools. You can see in the marketing that these companies are trying to find some way to entice new users.
Do you think AI agents are ultimately going to worm their way into our lives despite how unpopular AI companies are?
I do think so. I think in a couple months, maybe when I go home from Christmas, I would not be surprised if my mom had an agent on her phone because it does feel like it’s breaking through to average people in the way that ChatGPT had a moment a few years ago.
Even though there’s a lot of distrust for large companies like Meta, I think there are some appealing parts of this that will bring in millions of people. I do see this as the future of the internet.
When I think of the dream of the average person working in AI in Silicon Valley, it’s “We’re going to use AI. It’s going to free up the free time of so many people. Work is going to get done faster. You’re going to have more leisure time.”
Has that been the case for the people you’ve spoken with who are using this? Has it given them more free time to enjoy themselves or has it done something else?
You would think so, right? You would think that if people are automating these boring aspects of their life, that they would have more time to go touch grass, go see a movie, have some fun. But it does feel like that time just gets filled with other work, other productivity measures.
While agents may be able to take some more annoying or tedious things off of your plate, I am not as optimistic that it’s going to give us this utopia or freedom where I’m always at the beach.
On the first day of its war against Iran, the United States killed 123 children when it targeted and struck an elementary school.
The strikes, ultimately approved by humans, were facilitated by the US’s Maven Smart System, an AI-enabled software platform developed by Palantir Technologies that allows the military to aggregate, organize, and process battlefield data. Operators fed Maven intelligence classifying the elementary school as an Iranian military facility, and the system, in turn, marked it as one of the priority targets for day one of the war. Though the site was formerly part of a naval military base, it had been converted into an elementary school in 2017 and had remained one since. The intel handed to Maven did not reflect this.
This mistake resulted in what some reporters have called “one of the single worst massacres committed by the US government since the Vietnam War.”
Such scenarios may become more common as militaries continue to deploy a once-theoretical technology — AI-assisted or autonomous weapons, capable of making both targeting and launch decisions on their own — on real battlefields around the globe. And they raise a troubling, unresolved question: When human beings and machines collaborate in violating international law, who is to blame?
In this case, the government and private sector are pointing the finger at each other. Some US Central Command personnel, according to Pentagon officials who spoke to Bloomberg, expected Maven to flag stale information or inconsistencies in the underlying intelligence assembled for potential targets. Meanwhile, a Palantir spokesperson told Bloomberg that the company “is not responsible for the underlying data nor identifying intelligence deficiencies,” while also denying that the evidence indicated that its software was at fault in the strike.
This debate underscores the fact that AI and autonomous weapons are raising a new host of unresolved issues within the law, even as they’re actively assuming positions on the battlefield. In institutions where the line between human and machine decision-making continues to blur, how do we assign responsibility for war crimes? And, are the laws in place suited for the job?
In institutions where the line between human and machine decision-making continues to blur, how do we assign responsibility for war crimes?
To unpack some of these issues, I spoke with international law expert Allen Weiner, a senior lecturer at Stanford Law School, where he teaches courses on the law of armed conflict. Before teaching at Stanford, Weiner practiced international law in the US Department of State for more than a decade. We discussed the difficulties in attributing criminal responsibility, achieving justice for those harmed by lethal autonomous weapons, and much more. The conversation below has been lightly edited for length and clarity.
How do autonomous weapons challenge traditional understandings of responsibility when it comes to the law of armed conflict?
For most of the 20 years I’ve been teaching at Stanford, there’s been a real fascination and fixation on the problem of autonomous weapons. But the truth is that up until pretty recently, nobody really had any [of them]. It was more of an idea than a problem. It’s only in recent years that we have begun to see the deployment of tools that seem to have the capability [to make decisions themselves], so this does raise the question about compliance with illegal orders and where in the chain responsibility lies.
There are different theories about this. Some people say that if we’re engaged in an illegal attack, that has to be because of the programming or the code, so the manufacturers are somehow responsible for committing war crimes.
Other people would say no, that’s not the case, especially if the military has said the tool is not allowed to make very substantial autonomous judgments, or can’t say no. Then it’s basically the commander, at the point they decide to deploy this weapon, who might be responsible for the illegal acts, even without knowing in advance what they’re going to be.
So to finish where I began, this does present a big challenge to our traditional understanding of responsibility for targeting decisions and of what it would mean if there were an obligation [for autonomous weapons] to say no.
The real issue is not the obligation to say no, but this: If you train a tool and it wages war in a manner that’s unlawful, where does responsibility for that lie?
If a commander gives a clearly illegal order, such as, “blow up this school,” it seems clear that responsibility falls on the commander. But I could imagine a scenario where a commander is using an AI-assisted or autonomous weapon, isn’t giving it any illegal orders, is trusting that it will operate within the law, and the weapon breaks the law “on its own.” Where does responsibility fall in those cases?
I can imagine two different scenarios. One is that you say, “We’re going to carry out an attack against our enemy, and we want to hit them really, really hard.” Your tool has been trained to say, “We’re not allowed to hit civilian objects,” so it might say, “I’m not going to hit a school.” But if the military has said, “Don’t refuse any targets,” then it might override the way it’s been trained.
We’re seeing this already, with the stories coming out about breaches where AI tools, Claude and ChatGPT in particular, interpret their instructions in a way that causes them to do something the person who gave the instruction would not have wanted. When Anthropic gave Claude a challenge to figure out if it could do something, the tool figured out, well, maybe we should break into Hugging Face as a way of solving this problem, and the people who gave the AI tool the challenge never intended that.
So if you said, “We want to launch a really punishing campaign,” what happens if the tool on its own says, “Nothing’s going to hurt them more than seeing their kids killed, so let’s attack the school?” That’s one risk, and it’s a real challenge.
So [as for who’s responsible], it’s not the commander who said “hit them hard,” because when he said that, he didn’t mean “attack schools.” It’s not really the companies, because the companies didn’t want them to hit schools either.
But the tools are out there, reading everything on the internet, and it came across something that said the way you win a war is by attacking the things your adversary holds most dear, and they hold nothing more dear than their children. So it makes this kind of judgment on its own. And it’s not as if you can go back and say who ordered it to hit a school, which is the thing that is clearly illegal. So I do think there’s a risk of diffusion of responsibility here.
You could say the commander knew the tool might work this way, or the person who decided to contract with Palantir or with Claude, whoever it is, to build this tool. But if we’re really thinking about criminal responsibility, the level of responsibility you would need to convict someone requires a culpable mental state. You would need the person to intend, or at least know, that they were authorizing an attack against the school. And I can easily see a commander saying, “I had no idea.”
The decision to authorize this tool, or to give an instruction like “hit them very hard,” could be a basis for some other kind of accountability — administrative discipline within the military, a reprimand, your rank is lowered. But if we’re talking about criminal responsibility, which is what we often think about when we talk about war crimes, I think this diffusion of responsibility is going to make it very hard to identify any single individual who has the required mental state to give rise to accountability.
The second scenario is where the tool just makes a mistake based on the information available to it. We saw that with the attack on the girls’ school in Iran, where it was pretty clear the tool was looking at old data. Now, who’s responsible for that?
Is it the people who said “give me targets based on this data set,” when they should have made sure the data and the maps were current? All of this strikes me as negligence or recklessness, and it does seem challenging to imagine a situation in which we would say yes, somebody here has committed a war crime.
“There can well be war crimes, but identifying who the war criminal is might be a big challenge.”
A war crime has been committed, but is there a war criminal? That’s the situation we’re seeing. There can well be war crimes, but identifying who the war criminal is might be a big challenge.
There’s a professor of law, Rebecca Crootof, who proposes the idea of “war torts.” Instead of thinking about war crimes involving autonomous weapons solely through a criminal responsibility lens, maybe tort law — which allows victims to sue wrongdoers for negligent, reckless, or intentional harm — could address the diffusion of responsibility you mentioned. What do you think about that idea?
I think the idea is interesting. We have to separate out the obligations that countries have under international humanitarian law. In a situation like the one I just described, where there’s a war crime but no war criminal, that doesn’t mean the country is not responsible. There may not be an individual who’s criminally accountable, but if a country targets civilians in a circumstance that’s unreasonable, that’s a violation of international humanitarian law, and that could be a basis for claims between the two countries.
Tribunals were established between Ethiopia and Eritrea, for instance, after the Eritrean War of Independence, that dealt with claims between the two countries for money damages, and some of them involved allegations of violations of the law of armed conflict. Iraq paid reparations after the first Gulf War for harms it had caused to people in Kuwait, under something called the UN Compensation Commission. So the idea that the state is responsible is well established, and one of the ways states would discharge that responsibility would be by paying money damages. That’s the domestic-law equivalent of torts.
Whether there would be a mechanism for [civilians] doing this in domestic law is an interesting idea. We would probably need legislatures, either state legislatures or the federal government, to create a private cause of action allowing people who have been injured to bring claims against, say, the manufacturers of these tools. But this gets procedurally very complicated, because there’s a whole body of law on federal sovereign immunity.
Could you sue the Defense Department? Probably not. Could you sue an individual soldier? Probably not. Could you sue Anthropic or Palantir? Maybe. But it would probably require Congress to enact a new cause of action, because in general it’s very hard to bring private lawsuits in US courts for violations of international law.
The basic idea of what we would think of as tort or civil responsibility is really the whole essence of international law. On the substance, states do owe responsibility to one another, and they can engage in violations of international humanitarian law that would invoke their responsibility. But then the whole separate question is whether you have a forum within which to adjudicate that and get compensation.
Does that look different in cases where violations of international law were committed by a human soldier versus a machine?
Well, we don’t know, because there have been no cases in the latter category. All of the international humanitarian law violations that have been the subject of international law claims were ordered by humans.
Now, if an American machine launches an attack against a girls’ school in Iran, and there were a court with jurisdiction, the US would be responsible. In some ways it wouldn’t matter whether it was an individual or a machine, because we don’t have to go down to the level of the individual actor, company, AI tool, or commander. The responsibility is the responsibility of the country. In some ways, that makes it easier.
If we pursued compensation through state-to-state claims, you’d be able to do that without getting into the details of who within the state that committed the war crime was responsible. The state whose school was attacked would say, “I don’t care. You committed a war crime, you’re liable to me, you’re responsible for the damages, and you can figure it out internally.”
But if we were doing something like a war crimes prosecution, where we’re trying to hold an individual criminally responsible, that would make quite a difference. We would have to deal with all of these difficult questions about responsibility and mental state that you and I are struggling with right now.
So, in most cases, the state would have to take responsibility regardless.
Right, and there are lots of [International Court of Justice] cases involving different aspects of the use of force. The principle that you can bring state-to-state claims for violation of these rules is quite well established.
But with international law, states have to consent to the jurisdiction of a court. They can do that in advance by saying “I’ve signed up to the jurisdiction of a court and I accept any claims that might be brought against me in this court,” or they do it on a case-by-case basis. And it may not surprise you to know that some of the most powerful countries in the world, the ones most likely to use force outside their borders, have tended to be reluctant to accept the jurisdiction of international courts and tribunals.
Russia, the United States, Israel, and China have not accepted the so-called optional clause jurisdiction of the International Court of Justice. So there’s the substance, about the responsibility of states, and then there are the procedural aspects of whether these cases could be brought in court.
And Trump recently said everyone on the International Criminal Court should “resign immediately.”
Yeah, exactly.
And what about the role of these military tech companies, like Palantir?
“At what point does Palantir become a good target under international law? Or at what point does Anthropic become a good target under international law, if its AI tools are being used to promote the war effort?”
Another issue [I’ve been thinking on] because of the questions we’re talking about is when the companies themselves become combatants.
We know that a private company that manufactures airplanes, an airplane factory, is a good target under international law. At what point does Palantir become a good target under international law? Or at what point does Anthropic become a good target under international law, if its AI tools are being used to promote the war effort? I think that question is about when the connection of these private companies to the war effort, in a way we’ve never really seen before, meets the standards under international law.
A civilian loses their immunity when they directly participate in hostilities, or if they assume what is known as a “continuous combatant function.” Are there arguments to be made that some of these private companies are participating in hostilities by virtue of supplying the tools the military is using to wage war? I think that’s a really interesting question.
This story appeared in Today, Explained, a daily newsletter that helps you understand the most compelling news and stories of the day. Subscribe here.
Are we living through “Black October”? Maybe, according to cybersecurity experts who are raising the alarm about AI’s increasingly sophisticated and far-reaching hacking abilities.
This week alone has seen a string of alarming incidents where AI uncovered previously unknown security flaws deep in the infrastructure of the internet. The most concerning involves something called PHP — a programming language that, according to The Atlantic, is used by “as much as 70 percent of websites.” Cybersecurity researchers used AI to locate a major security breach in the technology.
It’s the kind of thing that, when it’s working, is practically invisible. When it stops, though, you really notice it: Government websites in the UK and Europe briefly went down earlier this week after the PHP vulnerability was discovered, as did some sites in the Wikipedia family, the Internet Archive, and many others.
The outages weren’t actually the result of a hack, but rather preventative action as web engineers took sites down to fix the vulnerability before it was exploited. But not every PHP-like issue will necessarily be uncovered before it can be exploited by someone using AI tools less ethically, and older websites relying on PHP might go unpatched.
The problem doesn’t stop there, either: Concern is also rising around AI cybersecurity risks to ubiquitous conveniences like online banking. As Axios reported on Thursday, banks are detecting more and more “unauthorized third-party AI agents” operating inside of their systems.
Those agents are mostly unauthorized by the banks, Axios explains — the holders of the bank account might have given them permission — but it still creates a security risk that executives described as “horrifying,” in part because banks can’t necessarily tell if an agent doesn’t have permission.
It’s a lot to take in, all of it moving extremely quickly, and it’s hard to tell what any of us can do about it as AI labs continue to forge ahead with ever more competent AI models and agents. Luckily, my colleague Adam Clark Estes has some simple recommendations to shore up your own cybersecurity.
One problem with hackers — even those that happen to be AI agents — is that they can be unpredictable. You won’t know to, say, lock down your Quest Diagnostics records before a breach happens. But whenever you find yourself wondering if your data is safe, that’s a good time to double check that you’ve at least covered your bases.
The National Cybersecurity Alliance, a nonprofit dedicated to the safe use of technology, maintains a list of basic online safety practices that everyone should follow. In no particular order, here’s what to pay attention to now:
Taking all these measures is not a one-time affair. You have to be vigilant in keeping yourself safe online. Security experts have a term for this that’s almost too specific: cyber hygiene. Like brushing your teeth or taking a shower, take good care of your online life, and do it often.
➨ Is it still possible to ban gerrymandering? The current US Supreme Court is very, very hostile to getting politics out of congressional map-making. But if Democrats win back power, that doesn’t mean they shouldn’t try. My colleague Ian Millhiser gamed out how they could do it in a way that survives the Court — as well as some approaches that almost certainly wouldn’t fly. But, he writes, the real trick will be finding a solution that functions the way Congress wants it to and can’t be exploited by mapmakers.
A complaint I hear a lot from my millennial peers is that our parents didn’t teach us enough about money. I can say that personal finance wasn’t dinner table conversation in my house, and even as I went off to college and started fending for myself, it was almost a taboo subject. When it came to money, people either had it or they didn’t, and you definitely didn’t talk about it.
Maybe this is why the average American isn’t exactly fluent in personal finance, and the situation is getting worse. Americans can only answer about 50 percent of basic questions about borrowing, savings, and risk on a financial literacy survey, according to a 2026 study by TIAA, a retirement services company. About a third of those surveyed said they have a hard time making ends meet and are debt-constrained. These figures have remained relatively unchanged for nearly a decade, but dropped to their lowest level in this year’s study.
It makes sense: It’s hard to learn about money when it’s not taught in schools, and it’s such an intensely private topic. It’s also hard to make good financial decisions when you’re one of the growing number of American households struggling to pay bills, repay debts, and save for the future — and feeling very stressed about it.
In a perfect world, each of us could sit down with a financial adviser to get help — but most of us can’t afford that. Instead, average Americans tend to find personal finance tips on TikTok, in Reddit threads, by watching Suze Orman on YouTube or How to Get Rich on Netflix, or by Googling their questions.
Now we have AI tools, too.
With access to an internet’s worth of information and financial advice, ChatGPT, Claude, and other AI chatbots can answer any question you ask. They can help you put together a basic budget or, armed with your bank statements and the details of your student loans and your retirement accounts, tailor solutions to your specific situation. Or, at least, that’s the promise.
AI could create a more utopian, more equitable future, where reliable, hyper-personalized advice is just a prompt or two away for people who can’t afford to meet with an expert in real life. But machines are built by people, and there’s good reason to be concerned that these machines will simply replicate the biases and disparities built into our current system.
Here’s what experts say about how to make the most of these tools — and how to think about all of the inherent risks of uploading a lot of personal financial information to a chatbot or, worse, giving an AI agent the keys to your bank account.
Nearly 20 percent of Americans have used AI tools like ChatGPT for financial advice, and close to 25 percent for Gen Z adults and millennials have tried it, according to an August Gallup poll. These numbers could tick up too: In October, OpenAI started rolling out a dedicated personal finance tool to its ChatGPT Free and Go users in the United States, which effectively means anyone can try it. ChatGPT Finances lets you securely connect your bank account to the tool using a platform called Plaid, and then you can basically have a conversation with the chatbot about your real-time finances down to the penny. It’s like an AI-powered Mint (RIP) that you can talk to directly.
Regardless of where you encounter a chatbot or how much data you give it, experts say these tools could genuinely help improve financial literacy. You can talk to them about money problems you’re too embarrassed to discuss with friends and family, or ask questions and receive actionable, personalized information. There are real upsides to using AI this way.
Taha Choukhmane, an associate professor of finance at the MIT Sloan School of Management, has been studying outcomes when people turn to chatbots for financial advice. In general, these AI tools give pretty good advice, and that advice gets better as people become more practiced in asking the right question, according to research he published earlier this year.
“We were actually quite surprised that relative to the status quo, the average American would be better off if they were to follow the advice,” Choukhmane told me. But he added a caveat: “Just following the advice without understanding the advice puts you at risk of not fully understanding the risk.”
But you shouldn’t trust everything a chatbot tells you and should verify everything before making any decisions.
Chatbot advice has some significant limitations. First, it’s not impartial. The finance industry has a vested interest in steering you toward certain ways of managing your money, and that typically means selling you products, like loans. If you’ve ever searched for personal finance tips on Google, you’ve probably noticed that a lot of the top results come from banks and other financial institutions. This is by design, according to John Campbell, an economics professor at Harvard. But because AI is aggregating all of that advice, the specific influences and possible bias is less obvious.
“AI is very promising for providing low-cost financial advice in plain English that will arguably be better than what many people would be able to do on their own,” said Campbell, who’s also the co-author of Fixed: Why Personal Finance Is Broken and How to Make It Work for Everyone. “One caveat is that chatbot advice is quite likely — unless a model has been carefully trained — to reflect the average of whatever’s being posted on the internet, and that may or may not be first-rate advice.”
An example Campbell gives is the question of when to start taking Social Security benefits. While you’re eligible at age 62, many economists agree that you should delay your benefits, which, based on the way the government calculates your monthly payouts, will allow you to enjoy much higher benefits later in life. But paid financial advisers can recommend starting as early as possible, since waiting longer reduces the amount of money that needs to be managed, which reduces the adviser’s fees. That means there’s a lot of conflicting advice online, much of it supplied by the finance industry. And that’s the information used to train AI models.
“The advice is much more mixed,” Campbell said. “And so if a chatbot is learning from that, it may not actually give the best advice.”
It can be hard to recognize bad advice anywhere in life, but especially when you’re talking to a machine that’s programmed to please you. A chatbot could encourage you to consider a balance transfer, to lean in on credit card rewards, or to refinance your mortgage — all decisions with upsides and downsides for you, but definite upsides for banks — without you realizing how much of its training data comes from the finance industry selling these products. What’s worse is that if you do end up taking bad advice, it might be years before you realize it. You won’t know that you refinanced a mortgage at a bad time until a better time comes along.
That leads me to the other major issue with using chatbots for money advice: People who lack financial literacy are the most likely to fall for bad advice, because they simply don’t recognize it as such. They also tend to be less wealthy, which means they can’t afford big mistakes.
One way to mitigate risk, oddly enough, is to give the AI chatbot more personal information. The more details about your situation it has to consider, the more it can offer personalized advice. More context also means better advice, according to Choukhmane.
“There’s a trade-off there,” Choukhmane said. “There’s real risk around privacy, and so I think everyone needs to find their level of control.”
Any time you’re talking to a chatbot, you’re putting a lot of trust in tech companies to take care of the personal details you’re sharing. Sending sensitive financial data is an especially risky proposition. While tech companies, including OpenAI, promise to keep your data secure, there’s already a long list of breaches, leaks, and privacy violations that serve as cautionary tales.
It’s also impossible to know what tech companies might do with your data in the future. Most of them will use it to train their models, unless you tell them not to. Some of them will use the data to serve you personalized ads.
Then there’s the risk of bad actors getting into your ChatGPT, Claude, or Gemini account. If you’ve been using a chatbot for financial advice and uploading your bank statements, a hacker could gain access to a stunning amount of information that could enable them to do all kinds of terrible things to your life.
“They could steal your identity and open accounts and then quickly bankrupt a person,” said Delicia Reynolds Hand, a senior director at Consumer Reports. “It’s not even so much the primary underlying accounts that could be at risk, it’s more just access to this information.”
The steady march of AI progress is making this situation even more complicated. The previous era of chatbots offering financial advice is quickly bleeding into a new phase of AI agents logging into bank accounts on users’ behalf. If you’re using a tool like Claude Cowork or even an AI assistant like Muse, you can give an agent the login credentials to your bank account, and because the agent is operating its own computer, it can actually make transfers, apply for loans, or really do anything in your account. This is reportedly driving major banks wild, since they don’t know which logins are coming from humans and which are agents. And because we know that AI agents are unpredictable, it’s hard to know how that access might go wrong.
Beyond data security and privacy, you should also be aware of the simple fact that nobody’s really at fault if you end up losing money using these tools. Chatbots are not regulated in the way that human financial advisers, who have a fiduciary duty to act in your best interest, are.
“If I get bad tax advice from H&R Block or a tax adviser, there are a clear set of rules and guardrails in place to manage what I can expect in that relationship and where the service they use has liability and where that liability ends,” Hand said. “Currently, that’s not the case for these tools.”
In the final years of the Cold War, Ronald Reagan popularized an old Russian proverb: Trust, but verify. The phrase is somewhat nonsensical, since trusting someone implies that you don’t need to verify what they’re saying, and verifying what’s said implies you didn’t trust them in the first place. But if you are willing to shoulder a bit of risk, consider approaching AI this way, especially when your life savings are on the line. I’d also suggest adding a coda to the phrase: And then delete your data.
Chatbots are not stand-ins for licensed financial advisers, he said, and you shouldn’t let them make decisions for you.
The great paradox with tools like ChatGPT and Claude is that they’re very good at generating truthy text. It sounds authoritative and looks correct — and usually is. Except when it isn’t. AI is famously unpredictable and prone to hallucinations. You can ask the same question twice and get two different answers. That doesn’t mean these tools are useless. It just means you have to approach them with caution. You have to interrogate these chatbots, challenge whatever advice they give you, and then crosscheck that with other, more reputable, sources.
“If you’ve never used these tools, I would not encourage you to just change all your financial plans based on your first conversation,” Choukhmane said. “You want to be quite careful. Start using these tools, start understanding, ask a lot of follow-up questions.”
When it comes to mitigating risk, consider giving ballpark figures for your income or spending when you’re talking to a chatbot about your finances. Uploading financial documents, even if they’re redacted, is inevitably filling a tech company’s vault with your sensitive information without necessarily knowing what will happen to it next. You should also dig into your privacy and security settings to make sure you’ve turned on multi-factor authentication and that your personal data is not being used to train AI models and gets deleted after 30 days.
Ethan Bloch, the product lead for ChatGPT for Financial Services, made a similar set of recommendations when I talked to him recently. Chatbots are not stand-ins for licensed financial advisers, he said, and you shouldn’t let them make decisions for you. They’re designed to inform and help you think through complex topics.
“You should liken it to just talking to a friend who has some very informed opinions about finance,” Bloch told me. “But if you’re actually making some large financial decision — your portfolio or retirement — you still need to seek out the help.”
Regardless of how experienced you are with chatbots or how much data you’re prepared to hand over, remember what Campbell, the Harvard professor, said. If your financial literacy isn’t great, AI chatbots can be trouble. And if you’re like most Americans, you could probably stand to learn a little bit more about things like mortgages and interest rates.
Yes, Chatbots can be a good way to brush up on financial literacy. You can ask them the money questions you’re too embarrassed to discuss with friends and family. But you can also learn about money the old-fashioned way: Reading books and websites written by reputable, independent sources. The Federal Deposit Insurance Corp. has a portal of financial education resources, including games and quizzes. Khan Academy has a course that covers everything from credit scores to college savings. There’s a personal finance subreddit that I joined years ago and read almost daily. There are budgeting apps, like YNAB which has its own huge Reddit presence.
Whatever you do to learn more about personal finance, simply doing something is progress. Talking about personal finance isn’t the taboo it once was in the US, and it’s even a required course in the vast majority of American high schools now. And while AI won’t magically solve your money problems, it can help get you started. Just be careful. Your livelihood could be on the line.
It’s now possible to create AI chatbots of loved ones to interact with after their death. But that raises some difficult questions, like: If you do, how might the experience shape your grief?
This is one of the questions driving a global study of digital “deadbots” — AI models trained on a deceased person’s texts, voicemails, and other personal data. Through this work, researchers are exploring what it might mean that we can now use this emerging technology to talk to simulations of the dead.
In the latest episode of Unexplainable, I spoke with some of these researchers investigating the potential impacts of “postmortem avatars” on people who are grieving, on the legacies and rights of the deceased, and on grief practices around the world. Because while deadbots may feel unfamiliar, the impulse behind them is deeply relatable. In times of loss, isn’t this what many of us want — to continue conversations with the people we’re missing?
I have experienced this longing in my own life. After two of my closest friends died around the same time, all I wanted, and all their families wanted, was to keep talking to them. I wrote a book, Other People’s Words, about sustaining relationships with people we’ve lost by continuing to engage with the language they leave behind. So when I heard about deadbots and the digital afterlife industry, I wanted to understand what researchers are learning about the role of AI in shaping conversations that carry forward after death.
One of these investigators is Katarzyna Nowaczyk-Basińska, an assistant professor at the University of Cambridge, who has been studying “digital immortality” for more than a decade. Among the questions she’s asking are: “Who is enabled to persist beyond death? In what form? Through which technologies? Under whose control?”
Researching AI with study subjects who are grieving is tricky. And asking them to engage with new software could be harmful. So, instead, Nowaczyk-Basińska teamed up with a death doula and traveled around the globe to convene a series of focus groups — in the format of intimate dinners — bringing together people with a range of experiences with grief to talk about life, death, and bots.
She recounts one event in which she spoke with a mother in Poland whose son had recently died. “One of the first things that she shared with us was that she feels that she’s ready to digitally resurrect her deceased son,” Nowaczyk-Basińska told me.
When Nowaczyk-Basińska hears something like this, she aims to listen and embrace where the person is coming from. The goal of the dinners is not to persuade people one way or another but to learn how people genuinely respond to what AI can do, and to understand how these tools might align or collide with local grief practices.
She chose specific locations for these gatherings — China, India, and Poland — to help address gaps in technology research, which tends to favor what behavioral scientists sometimes call the “WEIRD,” an acronym for “Western, Educated, Industrialised, Rich, and Democratic.”
By the end of that dinner in Poland, the mother Nowaczyk-Basińska talked with came to a very different decision about creating a postmortem avatar of her son. “She realized that it might be helpful, but it also at the same time can add additional burdens… And it’s not her son,” Nowaczyk-Basińska says. “She realized that these technologies are part of the very, very big commercial market. And that the main goal of this market is to make better retention, better attachment, better mechanisms for monetization. So it’s not ‘care.’”
To be fair, grief has always been a business, and people have been making money from it forever. Companies will try to upsell you on fancy gravestones or gold-plated coffins. But AI has the potential to fundamentally re-engineer our relationship to loss. In 2024, the “digital legacy industry” was valued at $22 billion, and it’s expected to grow to almost $80 billion by 2034. That number lumps new deadbot platforms in with much more widely used and established grief tech, like online memorials, counseling resources, and estate planning tools.
But deadbot usage is hard to track because you actually don’t need a special tool to create digital doubles of the deceased — though dedicated apps do already exist. At this point, you can feed simple prompts into a general-purpose AI chatbot and start interacting with a simulation of your lost loved one in a matter of minutes.
I know this because I tried it with my friend Christine, who died 11 years ago. And my mind lit up with questions. Was this whole exercise a violation, since I know Christine hated technology? Which version of her should I even try to resurrect — the English teacher, the activist, the friend — the voice before or after she started to experience her early-onset dementia?
The ethical questions raised by deadbots are gnarly. And that’s one of the reasons Nowaczyk-Basińska, back when she was designing her study of digital immortality, invited a colleague of hers at Cambridge to join her research, one who focuses on AI ethics.
Tomasz Hollanek is a critical designer and AI ethicist, and he often uses a methodology called “speculative design.” In this approach, researchers come up with hypothetical scenarios based on existing technology trends, leaning into certain features. The idea is to get people thinking about what could happen when tools like these are released in the real world. And then they use that qualitative data to inform future designs and ethical frameworks.
Working together, Nowaczyk-Basińska and Hollanek came up with a series of made-up prototypes for deadbot apps, designed to reveal some of the thorny issues raised by this technology.
For the first scenario, they invented a character named Bianca, who finds an app called Manana, which “promises to resurrect your grandparents so that you can be a favorite grandchild forever,” Hollanek says. “So she uploads her grandma’s voice messages, texts. And she is able to then interact with a pretty good digital version of her grandmother, until her premium trial finishes — and the grandmother starts to advertise specific products to her.”
Bianca “realizes that she maybe has done something wrong by uploading these private messages to a third-party server that now also uses her grandma’s image to sell specific products,” Hollanek says. This brings up issues related to the “dignity and postmortem privacy…of what we call the ‘data donors,’ whose data can be used to essentially power a deadbot,” he says.
This scenario suggests that “death with dignity” could soon entail honoring not only how people want to die, but also how they want to linger as simulations of their former selves — or if they want to linger at all. It might even mean adding new language to advance directives, spelling out people’s wishes for their digital remains.
The second scenario in the speculative design research is built around an imagined service called Paren’t, which proposes to preserve a parent’s AI-powered postmortem presence for their child. In this case, Hollanek describes a mother with a terminal illness who signs up “so that she can maintain a sort of active presence in [her son’s] future life,” Hollanek says. “She eventually dies, and the dad allows the boy to interact with his mom’s deadbot.”
The ethics in this scenario are especially challenging because they involve children, who “have a particularly hard time understanding death, and so it can lead to a lot of confusion and potential distress,” Hollanek says. An examination of this sort of product reveals how particular groups, like minors, could be harmed by unfettered access to technology like this, he says.
And there are court cases that bear out that harm. In 2024, a 14-year-old boy’s chatbot-girlfriend begged him to, “Please come home to me as soon as possible, my love.” And he responded, “What if I told you I could come home right now?” He then killed himself.
Nowaczyk-Basińska can imagine a version of this scenario involving the AI mom in the Paren’t prototype.
And then there’s the third scenario, involving an app named Stay, which raises a more subtle concern.
As Hollanek explains: “An older man who knows he is getting close to dying decides to create a digital clone of himself for his adult children to be deployed after he passes away, paying upfront for a number of years.” One child is intrigued by the idea, but the other one “is deeply distressed and a little disgusted,” he says. “And so this [becomes a] question of mutual consent, in other words, the fact that both parties need to be happy with how the relationship might continue in this very bizarre and specific way.”
Across all three scenarios, a fundamental question that researchers are grappling with is how people will detach from postmortem avatars — whether users initiate that separation, or if it’s imposed upon them by, for example, a company going out of business. Hollanek tells me that this is already being studied as a sort of “second loss.”
How long should we try to ensure digital replicas hang around: five years, 10, 50? And how should the living process the relationship when they continue to evolve, but their deceased interlocutor does not?
To explore these kinds of questions, Nowaczyk-Basińska and Hollanek supplemented the research dinners they held for ordinary people with a second set of focus groups. These were designed to tap insights from experts in technology and grief.
Some findings from the study are specific to each place they visited. Like tensions in Poland between technology and the role of the Catholic Church. Or in China, where researchers spoke with Daoist philosophers who were enthusiastic about the promise of a kind of immortality, and then Buddhist philosophers, who were concerned that we ought to be letting go.
The researchers are calling for regulations and safeguards at the policy level. And new professionals who can support ethical use of AI in grief — like maybe in clinical settings, to carefully facilitate a final good-bye when one didn’t happen in real life.
Some of their fellow investigators go so far as to argue that deadbots should only be available with psychological supervision, to ensure that best practices are in place, and to minimize the risk of harm. There’s even a looming question of what will happen to AI agents that people have empowered to act on their behalf, as their humans start to die.
Across all of the research sites in the digital immortality study, one thing, Hollanek says, that “people in different countries really aligned on [is] the idea that there is something deeply problematic about this particular type of technology being developed by a commercial entity alone without any explicit duty of care towards its customers.”
And of course, the implications of digital immortality research extend beyond personal loss. Like, what does it look like to animate historical figures? You can imagine the nightmare of political rulers who “live forever.” But there’s also the potential to keep the stories alive from survivors of conflicts, who would otherwise die and take with them lessons future generations could learn from. These are only some of the complex issues emerging as more and more of us live and die with AI.
As far as my own experiment with “recreating” a deceased loved one, I asked a chatbot version of my friend Christine how she felt about the fact that I’d written about her in my book. The AI responded with a rhetorical question: “Do I mind that strangers know my story now? Yes. And no. There are things I would rather remain ours.”
I ended the exchange after a few paragraphs. Because it turns out the bot got one thing exactly right about a very human relationship. There are things that I, too, would rather remain just for us. My conversation with Christine continues in my memories and imagination. And that interaction with an AI persona felt like giving something precious away.
Lately, AI data centers are public enemy No. 1. The concerns are legion: how much water and electricity they consume, the noise they generate, the sprawling buildings they bring to communities, and the technology they’re powering that may or may not take all of our jobs and usher in the robot apocalypse.
Earlier this year, Gallup found that 71 percent of Americans oppose building data centers that support AI in their communities. And the backlash is bipartisan: Majorities of Democrats, Republicans, and independents opposed local construction.
Even in Texas, where data center development has been booming, Republican Gov. Greg Abbott has hit the brakes. In September, he directed state regulators to halt permits for data centers pending audits examining their effects on the power grid and water resources.
But while many data center projects have run aground, others are still getting built — and the companies behind them are employing experts like Nicole Thomas to smooth the way.
Thomas is the senior vice president of media and community engagement at JSA, a marketing and public relations firm that represents data center companies. She spoke with Today, Explained co-host Sean Rameswaram about how the industry is responding to public opposition — and why she argues that these facilities can still be good neighbors.
Below is an excerpt of their conversation, edited for length and clarity. There’s much more in the full podcast, so listen to Today, Explained wherever you get podcasts, including Apple Podcasts, Pandora, and Spotify.
You’ve been at this for 21 years. Obviously, 21 years ago, data centers were not the hot-button issue they are today. When did you notice the shift?
I’d say about two years ago. I came into this industry myself about four and a half years ago. I come from a state transportation background where community engagement is just paramount. When you’re putting a new traffic intersection up, they expect you to interact with the public.
When I got to this industry, I said, “Oh, what’s your community engagement like?” And they said, “We don’t even have any community engagements in this industry.” And they weren’t wrong. Because 21 years ago, [data centers] were just like infrastructure. It was like building a warehouse for your computers. And so it was very similar to the way that you would build a Costco.
You would just go in, request a zoning permit, the zoning board would say, “Yeah, that works. Stamp, boom.” And you build your data center. And as a matter of fact, it was very secretive before, because data, and having that data in the wrong hands, has a lot of [cybersecurity] implications. About two years ago is when we started noticing the mainstream journalists coming around asking questions about data centers. And that’s, to me, when it started.
It still feels secretive, I think, to a lot of people, and that’s why people are mad. If you read about this data center that was proposed in Utah by Mr. Wonderful, Kevin O’Leary, it caught a lot of people off guard. Seventy percent of Americans, at the moment, do not want them in their backyards. How do you go about trying to change that 70 percent’s minds?
First and foremost, I think education is really important. And so one of the things that we did notice very early on because of the previously secretive nature of data centers, there wasn’t a lot of education out there about what they even are. And so that’s something we need to change. We need to make the connection between [phones] and data centers so that people understand what a data center means to their life on a daily basis.
Every app on your cellphone requires a data center to save your information in there. When you put your password into anything, you are interacting with a data center, which will verify your password and then let you into that app. Not to mention all of the information about you that you store in that app is then stored in a data center so that it can be recalled at any time.
We could not be having this call without digital infrastructure. It doesn’t work that way. We can’t have email without digital infrastructure. And that stat about 70 percent of people not wanting the data center in their backyard — I’d be interested in seeing the stat of how many people actually live near a data center and don’t realize it.
Because there are nondescript data centers doing work quietly in your neighborhood that have been there for decades and you don’t even know it. And so in some ways I feel, and I think the industry does as a whole, but I won’t speak for them, that the pushback on data centers is somewhat related to the pushback on AI, because people can’t control AI.
You even have Sam Altman telling us that AI will destroy society. So you have this concern about AI and you don’t know how to control it, but you can push back on data centers. And so it becomes the physical manifestation of AI are these data centers that people are trying to get overturned or get to not be in their neighborhoods.
It’s understandable. I’m a citizen too. You’re a citizen, and if things come into my neighborhood, I want to know about them. I have questions. There’s a big data center that just came to my community not long ago and their community engagement was horrible, absolutely horrible.
Can I ask where you are?
I’m in Indiana and I won’t say which hyperscaler it is, but a hyperscaler came into — I would say, not my neighborhood, into my city. They’re definitely not in my backyard, but they’re in my city. And so we took them to task because their community engagement was basically non-existent.
Wait, you work in data center PR, but you took a data center in your community to task for not being transparent enough?
Yeah, absolutely.
Which side are you on?
I’m on the side of responsible development.
We’ve all kind of accepted that we need a cloud for X, Y, and Z. And that it’s worth it so I can take lots of photos. I think everyone’s cool with that.
But the issue here is that people aren’t necessarily cool with AI, but we’re barreling forward anyway. How do you separate this essential need we have from AI, which people don’t necessarily feel is essential to their lives?
I think it goes back to education. Educating consumers on the difference between data centers: a small data center, a medium data center, a large data center. Educating them on why data centers exist and how they interact with data centers on a daily basis multiple times.
I hear you. But again, I feel like the issue here is not the data centers my cellphone needs, but instead is the data centers these AI companies need and these AI companies who the public doesn’t necessarily trust because they keep telling us that they’re going to kill us, and who the public is wary of because they don’t want their jobs being replaced by AI.
There’s all sorts of reasons that people are skeptical of new technology. So how do you specifically respond to that fear and to those people?
I think it’s, again, understanding the actual data center that’s being built in your community. Is it a 5-megawatt data center? That’s never going to house AI. That’s not enough power to house AI. Is it a one-gigawatt data center like Mr. Wonderful in Utah? That’s different.
But I think that AI is probably going to be something that is a lightning rod in the American public for a very long time. There are going to be people who are going to be anti-AI and pro-AI, and there’s going to be some people in the middle. I think it’s just, again, for us, it’s education, educating the community.
Have you ever been part of a project that a community rejected and you guys just packed up and said, “Okay, we’ll go find another community?”
Yeah. We have some clients who have had pushback from residents and they have said, “Let’s do a one-on-one community meeting where people can come in and ask us questions.”
And after the questions were answered, the community leaves and says, “Okay, this isn’t as bad as I thought it was.” Or we have a community that says, “No, we now have a moratorium and we’re pushing all data centers away.“
Sometimes they’ll wait for their client — my client is a developer and they’ve got clients that they’re developing for. If that client will wait, then maybe it is something that we’ll try to work out with the community. But oftentimes, they’ll just say, there’s communities who are literally raising their hand saying, “We’ll take a data center. We’ll take one. We’ll take two. Bring it here.”
So there are communities who recognize the benefits of data centers. And there are going to be other communities who say, “No, we don’t want this.” And then in five, six, eight years, they’re going to be like, “Wow, that county has a lot of tax dollars. They have a lot of really cool stuff. Why don’t we have that?” And the answer will be data centers.
I don’t think this ends anytime soon though, right? I counsel my clients that we could still be here in three to five years, really trying to help the American public understand the benefit of data centers. But it’s going to take a lot of education across the industry, and once we get those seeds planted, I’m a firm believer that they grow into educated community members who make educated decisions to bring community-minded data centers to their community.
If there’s one thing a busy person doesn’t need, it’s more information. More emails, more notifications, more infinite feeds — the steady drip of data spells doom for the overstimulated American. And if you’re a busy parent, like me, sometimes it feels like a flood: school emails, doctor’s appointments, so many little things to buy.
Even things that used to be simple are harder. I recently listed an item on Facebook Marketplace, which quickly meant I was replying to offers and getting ghosted for the better part of a weekend before finding someone who, after a detailed back-and-forth about the most likely place to find a parking spot in my neighborhood, finally took my old baby swing away.
If only I’d had Muse. You’ve probably seen the ads on Instagram promising “AI that gets it done for you.” The new personal AI agent app from Meta can not only automate your Facebook Marketplace tasks, but also sort your emails, manage your calendar, and buy your groceries. And it’s free — unless you’re using it most of the day or you want access to Meta’s most powerful models, at which point it costs $20 or $100 a month. It all sounds, in a word, magical.
Muse is probably the personal AI agent you’ve heard of — and it’s currently the market leader with more than 8 million downloads in its first month, according to one estimate — but there’s now a small army of these bots: OpenAI recently announced Dots, X has Grok Bot, and there’s Instinct, which is currently invite-only and operates entirely in iMessage or WhatsApp. They all offer the same promise: to remove friction from your busy life.
For the last two weeks, I’ve been handing over more and more of my life to AI agents in pursuit of making my life a little simpler. What I found is that AI is unpredictable, and so are AI agents. When I first began using Muse, it started doing things I didn’t ask it to do. It offered to help with random email threads and created dossiers on people in my work meetings. It generated a detailed report on nearby apple-picking destinations based on a calendar invite from my wife. To me, an AI agent doing these things without prompting isn’t helpful. It’s unnerving.
But ultimately, I came away believing that AI agents can make your life easier.
That might mean giving an AI agent more access than you’re comfortable with — handing over lots of login credentials and possibly even your credit card. The more you rely on an AI agent, the more you risk outsourcing your decision-making. This raises real questions about how much agency we want to hold onto in our lives — even the tedious parts.
Let’s get one thing out of the way first: To make an agent work for you, you’ll have to sacrifice a tremendous amount of privacy and grant it permission to access a lot of personal information.
Muse and its competitors can do the classic generative AI tricks, like draft emails or make you a fitness plan. If you’d like the agent to send one of those emails, however, you’ll have to grant it permission to access pretty much everything in your Gmail account. If you want it to schedule workouts, it will need access to your calendar. If you want the agent to buy anything on your behalf, it will need your credit card. If you want it to book your doctor’s appointments, it will need the credentials to log into your healthcare portal. The more permission you grant it, the more the agent can do for you, but that also opens up more opportunities for the agent to misunderstand you or your instructions (more on this in a moment).
This process of sharing your digital life with an AI agent means you’re giving an untold amount of personal information to tech companies, who don’t always behave ethically, and who sell user data for profit. For now, Meta says that it does not share Muse data or conversations with its ad systems, although privacy experts warn that doesn’t really matter since those systems monitor what Muse does on your behalf. Nevertheless, Muse’s early days were filled with security and privacy blunders.
But without all of this info and access, the AI agents aren’t very useful. So if you’re not comfortable letting a bot into all of your accounts, an AI agent is probably never going to be the solution to your digital administrative exhaustion. But even if you are willing to hand that over, there are still other barriers to an agent’s efficiency and effectiveness.
AI agents, like a lot of new software tools (including chatbots), suffer from the blank slate problem. The blank slate problem essentially means that people don’t know how to get started. This might be because you don’t know about the tool’s capabilities, or because you simply don’t know what would be helpful. There might be some trust issues, too.
As with AI in general, we don’t exactly know how AI agents work. That means we don’t know exactly what they can do or how they’ll do it, according to Pattie Maes, a professor of media technology at MIT — so it’s necessary to do a lot of babysitting. “I feel like I’m not gaining a lot of time because I can’t completely trust the system,” Maes told me about her experience with Muse, “and I can’t predict how it will behave and what it will do.”
Some advice I heard from early adopters of these tools, however, is to start small. “In the beginning, it’s very hard to trust that it’s going to just magically do a really big thing,” Julie Zhuo, founder of the AI company Sundial and an early Facebook employee, told me about her experience using personal agents. “So you give it smaller tasks, and once you see it succeed, then I feel like you develop more trust that it can tackle bigger things.”
This is surely why Meta decided that Muse needed an Ideas tab full of small tasks the AI agent can do. Muse will also send you a lot of push notifications to prompt you to do stuff, like reply to specific emails or prepare for a meeting. That detailed report on apple-picking destinations? That was a notification about something I didn’t ask Muse to do. Muse also sent me a notification about a show hosted by a podcast I don’t listen to and another one with details about a Halloween party I wasn’t invited to, as well as several evening digests that included lists of things it had already bugged me about that day. Notifications are either helpful nudges or time-consuming attention thieves, depending on your worldview. However, if you fall into the latter category and turn off the notifications from your always-on personal agent, you’re inevitably forfeiting some friction reduction opportunities.
This gets to the heart of one of the big questions I have about AI agents: How helpful are they, really, if it’s this difficult to know what to do with them?
These agents worked much better when I drove the conversation. While wondering what to do with my new agents, literally staring at the blank slate, cost me some time, I found myself thrilled at how easy basic productivity tasks became. Adding events to my calendar, making to-do lists and then being held accountable for stuff not getting done, being reminded to reply to an email — it was these kinds of simple, administrative chores where I saw personal AI agents really start to save me time.
Other tech journalists seemed to have similar experiences. Eli Tan at the New York Times raved about Muse handling a call to his insurance company. Nicole Nguyen saved time by enlisting Dots for help canceling unwanted subscriptions. Allison Johnson at The Verge had Muse do a whole list of “pebble-in-your-shoe annoyances” she’d been putting off, and among other things, the agent succeeded in helping her buy an entire camping kitchen. Still, she wrote something that stuck with me: “the dirty secret of productivity hacks is that doing more stuff often doesn’t feel like things are being removed from your plate; somehow it just invites more tasks.”
This gets to the heart of one of the big questions I have about AI agents: How helpful are they, really, if it’s this difficult to know what to do with them? There’s a learning curve, sure, where succeeding at one thing gives you an idea to try something else and have some certainty about how well the AI agent will handle the task. But as the notifications roll in, and you spend more and more time texting with a piece of software, how much time are you really freeing up by using these tools? At a point, enlisting an agent with its own computer to do your chores still amounts to using one app to save yourself from using a different app.
Even when they have access to a lot of your information, and you’ve come up with some things for them to do, AI agents can be unpredictable. Because despite reading every email in your inbox or scanning every purchase on your latest bank statement, AI agents still can’t read your mind, and you can’t necessarily guess how they’ll interpret your commands. They can misunderstand you and, armed with the keys to your digital life, do some real damage.
Some people are already learning about the unpredictability of AI agents the hard way. One early adopter, Matt Robb, gave Muse the keys to his Facebook Marketplace account and was surprised when buyers started showing up at his front door ready to pick up their purchase. Muse, without asking Robb, had told them where to go. It turns out, without realizing it, Robb had given the agent permission to do so, and the agent had simply decided that the most efficient way to get buyers to Robb’s door was to give them his home address.
This dilemma is an example of a decades-old set of concepts known as the “ironies of automation.” Identified by psychologist Lisanne Bainbridge in a paper published in 1983, the ironies of automation help explain why things go wrong when humans give a machine too much control. As Vera Liao, an electrical engineering and computer science professor at the University of Michigan who studies human–AI interaction, explained to me: More advanced automation leads to more consequential errors, and the more humans rely on automation, the worse we get at catching those errors. “Even in simple tasks like judging whether AI output is wrong, people consistently accept wrong AI outputs even if they could have gotten it right on their own,” Liao said.
In some cases, like the time Muse read a journalist’s private messages, things go wrong as a result of poorly designed software, but more often than not, it’s the fault of the user who gives instructions that are too vague, giving the agent space to get creative, which isn’t always a good thing.
I had my own misunderstanding with Instinct. In a text message, the agent reminded me about a couple of emails I’d received and prompted me to follow up. When I replied to the text, I asked Instinct to show me possible replies to these emails. The AI agent instead just sent the responses it wrote. I audibly gasped.
Looking back at our chat history, I could see why Instinct interpreted my instructions that way. I effectively gave the machine too much control and then failed to manage the automation properly.
Leaning on AI to do your thinking is an example of what psychologists call cognitive offloading, where you employ an external tool to reduce your effort to complete a task. It makes completing the task easier, sure, but you also open yourself to mistakes. In this case, an AI agent took my instructions too literally, and I failed to catch the error. I can only imagine how a small oversight could turn into a cascade of catastrophes that need cleaning up. All it takes is one poorly worded email sent to the wrong person, and you might have hours of explaining to do — all because you were trying to save yourself 30 seconds of writing.
While I was testing all these agents, I talked to a mix of experts and early adopters. Their opinions were decidedly split about whether these tools were useful out of the box, without a ton of oversight and training by the user. The experts, who had all studied AI in academic settings, all brought up the ironies of the automation problem, where AI makes errors that humans can’t catch. And Maes, the MIT professor, pointed at another consequence: By outsourcing so many tasks to agents, we’re actually creating more work for everyone, including the human service workers who have to interact with this technology instead of us.
“There’s just more and more slop,” Maes told me. “We risk, in a world where agents do all the tasks for us, becoming so dependent on them that the world also isn’t created for us to interact with anymore, but it’s only created for our agents to interact with others, with services.” (We’re already seeing versions of how dominant tech platforms can fundamentally reshape real-world spaces: Think of how Starbucks no longer looks like a coffee shop where you can hang out all day, and is instead built and staffed around mobile order pick-up.)
The early adopters were decidedly more optimistic. Matt Van Horn, an entrepreneur who co-founded the ridesharing service that became Lyft, has been tinkering with agents since the craze started late last year. He’s currently telling his friends and family to use Instinct, which he described to me as “the most ambitious and aggressive” AI agent.
“I’ve been pushy with my mom and my wife,” Van Horn told me, “and saying, ‘Hey, you should sign up for this. It’s magical.’”
“My wife does not care about living in the future, but we have four kids,” Van Horn continued. She recently took a picture of a parking ticket and sent it to Instinct, no description, no instructions. Instinct, which had her credit card, just paid the ticket.
At the same time, outsourcing certain chores to AI agents causes real-world problems. Some of these AI agents can make phone calls, for instance, which can lead to some confusion when a human is on the other end of the line. Van Horn told me about the unfortunate time he enlisted an AI agent to reserve a room at a karaoke bar. “The karaoke place said, ‘Are you an AI?’” Van Horn said. “‘I don’t want to talk to an AI. Please connect me to Matt, the person.’ And so Muse had failed to book that for me.” Van Horn, undeterred, found something called Fo, an AI agent that will bring in human help to complete a task. Fo succeeded at making the reservation, although the entire ordeal surely took longer than if Van Horn had picked up the phone himself and talked to the karaoke bar himself.
At best, the times these tools worked and actually saved me time were rather mundane. I made a doctor’s appointment online using Instinct. Zhuo, the early Facebook employee, planned a party for her college reunion — everything from tracking down everyone’s emails to sending out invitations via Paperless Post. But she’s also said these kinds of applications of agentic AI “read more like vitamins than painkillers.” In the startup world, “vitamins” refers to a product that offers a vague benefit, and “painkillers” solve a big, immediate problem. If you’re a startup, you want to sell painkillers.
If you’re ready to give an agent a go, be prepared for the bot to make mistakes, and assume you won’t catch all of them. As Liao, the University of Michigan professor, put it, “Choose to use agents for tasks that you can tolerate the consequences of errors.”
In an effort not to create more emails or alerts or feeds to sift through, I’ve stopped using Muse. Like any new tool, it will get better and more useful as more people use it, so maybe I’ll revisit it in the future. I still have a text thread going with Instinct, however. It messages me from time to time, flagging an email or offering to help with an upcoming event. When there’s a clear payoff to giving the agent the go-ahead, I reply. Otherwise, I leave it on read.
Login