© Jason Henry for The New York Times
It feels like the skies above our AI-frenzied world have darkened in the past month. Employees at Anthropic and Google started quitting, citing fears that the AI was spiraling out of control. After a string of agents started hacking websites, researchers used AI to discover vulnerabilities in critical systems, leading to large swaths of the web going dark last week and giving rise to a scary term: Black October. At the same time, as it continued to ramp up AI spending, Microsoft offered buyouts to 60,000 employees — over a quarter of its workforce — and economists once again warned of a bubble.
Yet, a lot has happened recently to make AI genuinely useful for more people. AI agents are getting better and can now make restaurant reservations for you. Walmart made it possible for agents to shop on behalf of humans, a bold move since Amazon banned the practice. Apple announced an AI-ready home display that can turn on your lights. And let’s not forget that AI is also helping scientists find cures for formerly incurable diseases and mathematicians solve previously impossible problems.
It’s hard to reconcile the incongruity between the recent bad news and good news. For most of us, the state of AI is anxiety-inducing — especially when it comes to our livelihoods. According to a Pew Research Center study published in August, 71 percent of Americans think AI will lead to fewer jobs in the next two decades. That figure is up 7 percentage points from where it was two years ago. Sure, in the best-case scenarios, AI could erase inefficiencies at work, streamline our daily lives, cure diseases, and generally improve the human condition.
But what if you are the “inefficiency” at work that gets erased?
That tension is at the heart of Ethan Mollick’s latest book Co-Existence: The Next Phase of AI. If there is such a thing as an AI sage, it would be Mollick, an associate professor at the Wharton School and faculty director of its Generative AI Labs. To the nearly half a million subscribers to his Substack newsletter, One Useful Thing, he delivers semiweekly updates from the jagged frontier of the technology’s development. His message in any medium is resoundingly clear: You need to use AI in order to understand it. And to get the most out of using it, he argues that you have to accept that AI is not going away; it’s only going to become more important.
That ethos doesn’t make the existential threats of AI any less daunting, but that’s not the point. Unless you’re an early Anthropic employee or a United States senator, there’s probably not much you can do directly to rein in AI’s potential harms. But you can, according to Mollick’s worldview, decide how the technology could fit into your life and think about how you fit into an AI-dominated future.
I recently spoke with Mollick about his new book, what life and work might look like in the agentic AI era, and what it means to be an artisan when even knowledge is industrialized. His theories about living with AI are meant to help you navigate the future, regardless of the latest forecast. This interview has been lightly edited for length and clarity.
We’re entering a new era for AI, and the vibes are different. AI agents are something that many of us will encounter, interact with, and maybe even embrace. I’ve always known you as someone who jumps on the cutting edge of this technology to figure out what it can do. What do you think most people don’t understand about the next generation of AI?
I really think people haven’t internalized what agents do. Agents mean many things, and we can talk about personal agents, like Muse. But the fact that the problems that plagued working with AI agents before — you had to guide them each step of the way, they made mistakes, hallucinated constantly — that’s kind of gone. You can regularly get hours, days, weeks of work out of a system.
It’s not just a theoretical thing anymore. I talk to coders all the time who are a hundred times or more productive than they were before. And in other fields, we’ve kind of moved past the theoretical discussion about AI being, “This is going to be a big deal,” to, “It’s here.” From a whole kind of co-intelligence perspective, we’ve gone from, “Here’s how we use AI and work with a chatbot back and forth,” to, “These systems are now doing real work in ways that matter to real people and are reshaping real fields for better or worse.” And if you’re used to using AI a year ago, if you’re not paying for one of these systems, it’s really easy to have lost track of what’s going on.
What about the practical, everyday ways that AI is disrupting our lives? And what do you think are the limits of that disruption?
So outside of the job world and homework cheating, I think life has changed only a little bit for people, partially because these systems were kind of complicated to use and you had to decide what you were using them for. But I think the advent of personal agents, like Muse and Dot, are going to lead to a very rapid change in a lot of ways.
Most people I know who aren’t experts who use these systems, like my parents, are using these things spontaneously. They’re finding that once you know how to prompt the AI to do something, it can do things spontaneously and come back to you because it has your email and everything else. Just as one example, I was unable to figure out how to cancel my standing $8-a-month wifi subscription from my airline, and I was like, figure this out. And it went to customer service, joined chats, then I have no idea what it did, but now I don’t have a subscription to wifi anymore, thank goodness.
“I deeply worry that if I de-skill myself entirely, I won’t even know how computers work.”
So there is this sort of lowering friction piece that’s super interesting, with good and bad elements to it. The bad side of lowering friction is both that every article gets written by AI, so there’s no human thinking in it anymore, and that companies don’t want 10,000 customer service agents coming to them demanding new things. So I think we’re just on the edge of seeing the mass change in everyday life. Right now, AI is a really good second opinion on everything. As agents become more popular, they start to take action.
How do you know when to let AI do the work, though? We know that offloading certain tasks to AI can make you worse at doing those kinds of tasks in the future. How do you take advantage of AI’s ability to do drudgery but also hold onto your own expertise?
If there’s a personal theme underlying Co-Existence, it’s that one. That is an incredibly hard choice to make, and we’re historically very bad at making it.
Students are the No. 1 case because they’re willing to offload homework because homework sucks and because the incentives for doing any piece of homework are terrible. But overall we need to do it to learn. So people offload that, but even if you’re trying to do the right kind of thing, like using AI only to help you learn and not to replace learning, it turns out that when chatbots are in assistant mode, as opposed to using them as a tutor, it feels like you’re learning even though you’re not. You feel like you’re in the loop even though you’re not. And so that assumes that we both know we want to offload and we do it intelligently and that we do it in a way that helps our long-term gain.
I am worried about us being able to do that. You have to draw lines for yourself about what skills matter to you or not, and you have to be thoughtful, in a way that we’re usually not, about what we’re giving up and what we’re not. Technology’s designed for us to give up stuff. We don’t walk 50 miles a day anymore in forced marches. I don’t know how to weave clothes. There’s a lot of stuff I’m very happy to not know: how to maintain my computer in some ways, even though that’s dangerous. I let Codex scan for security issues because it does a better job than me faster. But I deeply worry that if I de-skill myself entirely, I won’t even know how computers work.
So I don’t have an easy answer, but I think that that will be the defining question for most people in their career paths right now and their lives: What do I give up? I think we can be too sentimental about things that we’re giving up. It’s okay, but we don’t know the full chain of consequences. I think that struggle is going to be central to a lot of people’s experience. I wish I had an easy answer, but now that AI can do so many things, de-skilling is a real problem at this point.
We’re living through a huge transformation. That’s increasingly clear. Besides losing their jobs, a lot of people are scared of losing the sense of what makes them human and what makes them valuable. How do people hold onto that in the years to come?
There’s a transformation happening, and for some people, the work will get more satisfying. There’s a lot of coders who did not like writing lines of code, but they did like summoning software and figuring out what they wanted to build. Their lives are much better because the AI does all the tedious work. They can command things to be built, and it’s so much more exciting than it was before. It lets people who didn’t code start to do incredible coding projects. So on one hand, that’s deeply freeing of the kinds of work that we did. On the other hand, there are people who love writing those lines of code and that’s gone now. You’re a hand-weaver in a world of power looms. Your stuff might be elegant and beautiful, but it is not going to last in a world where industrialization happens.
Industrialization versus craft is a way to think about this. There’s a lot of craft that we’ve grown to really appreciate, like the handcrafted sentence. Those things will be under threat from the logic of industrialization, which is that we need to do more of it. I think that there’s going to be a shift. There’s satisfying work to do, but I think a lot of what was satisfying human work will be under pressure to disappear, because you took pleasure in something that was not automatable, but now is.
We’ve been through this in industrial revolutions before. This is happening at a very large scale to intellectual work, where craft was a very important part of what we did. Mathematics was craft, software engineering was craft. I think that we’ll be under pressure to produce more of it, which is what industrialization’s about, even if that destroys the value of doing this as a person. You get paid to do it now, but you’re not getting the same value you did before.
“We’re no longer in a place where AI is going away.”
I think we’re going to find lots of crises of meaning. Some people are going to find huge amounts more meaning in their work, either because the work itself is now much more interesting. Work is a lot of crap, too. We can say great things about it, but people spend a quarter of their work life saying they’re bored. It’s not like this was a great job for everybody. So some people will have much more excitement in craft. Some people will have it extracted out of their job, and it’s all going to be happening at once. I know it’s not a satisfying answer, but that’s what’s happening.
Forgive me for saying so, but that was an anxiety-inducing answer. It’s an anxiety-inducing time. I wonder what advice you have for us anxious people?
I think anxiety is real, and there’s different levels of anxiety. We’re going through a wrenching change, or we will be going through one that we’re on the edges of now. There is also existential risk about whether AI will kill us all, where we clearly need more regulation. The answers are unclear. Smart people seem to think that that’s a real danger. It’s hard to know. But leaving that aside, there’s anxiety about our own transformation, our jobs. I mean, the industrial revolutions worked out great in the end, but during it, [that] was the time of Marx and anarchists and Dickens writing about London. There’s a transition we have to go through here.
I don’t think we can pretend it’s not happening, and I see a lot of people taking comfort in the idea that maybe this all will collapse and AI will go away. Even if there was a bubble financially, we’re no longer in a place where AI is going away or stopping development. I worry a lot about people doing that. Once you face that, though, I find people often reconstruct meaning in very exciting ways. Thinking about what more you can do and where you find joy in this is actually quite exciting. So I think that anxious optimism is a reasonable place to be.
What I worry about is sticking your head in the sand saying, “I’m not going to learn AI. This will go away. I’m going to pretend it’s not happening.” I mean, I wish I could give you that option. I think there’s a lot of people who wish we could slow things down or turn a dial down, but there’s also a lot of exciting things. There’s a lot of hard problems in the world that are unsolved.
If you lean into the positive pieces, you can ask how we mitigate risk but also think about how we can build a better world. Part of what makes me nervous is people sitting out that second question and just filling up with anxiety rather than believing we have a tool that can rebuild things. How do we rebuild a better world for journalism or for nonprofits or whatever it is? I think it’s a reasonable question to ask, and I worry it’s asked too little.
© Adriano Machado/Reuters
© Athit Perawongmetha/Reuters
© Victor Llorente for The New York Times
We’re living through an AI-fueled “Black October,” and things are changing fast. Hackers are using AI to breach countless companies. AI tools have uncovered security flaws that could leave 70 percent of all websites vulnerable. Governments are scrambling to secure their sites. Banks, too. And because cyberhacks can hurt supply chains, some longtime AI watchers are going full prepper, stocking up on food, cash, and more.
In the center of this story is Anthropic, one of the world’s leading AI companies, which was explicitly founded on the promise that it would be a “good guy” company in a world of dubious tech giants.
Anthropic says all the latest developments are very serious. Its security chief wrote last week that things are “going to get a lot worse before they get better.” Two weeks ago the company warned that AI models could soon self-replicate on the internet-connected devices in your home, like smart fridges and robot vacuums. And its co-founder now says “the singularity” — when AI systems speed up technical progress in a way that’s hard for humans to comprehend — “is here.”
You might think, given its view of the risks, that Anthropic would be pumping the brakes on its cutting-edge AI progress. Just the opposite. Anthropic filed for a blockbuster initial public offering on Monday, chasing a valuation of $2 trillion even as it warns in the IPO filing that AI could “pose catastrophic or existential risks to humanity.” Every day, it is racing toward recursive self-improvement (RSI), where AI models are smart enough to build the next, even smarter AI models by themselves.
How does Anthropic justify that?
As its CEO, Dario Amodei, reminded the world last week, he built the company on one big bet: that instead of driving a race to the bottom, where companies skimp on safety in their rush to beat competitors in the market, Anthropic could actually catalyze a “race to the top.”
As I’ve watched Anthropic double down on its “race to the top” strategy, I’ve been uncomfortably reminded of another very risky gambit: Sam Bankman-Fried’s crypto fraud of 2022. There are important differences between the two; this is not a perfect parallel. But both involve a tiny handful of figures, strongly influenced by effective altruism, making huge bets on behalf of many, many other people.
EA culture prides itself on offering a counter to commonsense morality. The community has shown a willingness to take big risks and justify them with complex calculations. Sometimes that pays off. But believing you are intelligent enough to adopt a counter-intuitive, galaxy-brained strategy and end up saving the world with it often involves some hubris. And hubris makes me nervous in a field as high-stakes as AI.
The idea is that if Anthropic proves it can build AI that’s both powerful and safe, the market will expect both — so other AI companies will then feel compelled to compete on safety, too. That may sound nice in theory, but it means Anthropic has to keep building ever-more-powerful AIs.
“The forces that drive the race to the top are hugely more effective if the company practicing them is at the frontier,” Amodei wrote. “And looking at that strategy as a whole, I’m proud of our track record and stand by it.”
In other words, Amodei is sticking to his bet.
But Amodei is gambling with, by his and his company’s own telling, the future of humanity. If AI goes wrong, it could lead to our gradual disempowerment, a galling concentration of power, or even complete extinction.
So there’s a very basic question that the public deserves a good answer to right at this pivotal moment: What is the evidence that Anthropic’s stated strategy — compete at the frontier so as to drive a “race to the top” — is actually successful overall?
From the outside, it looks like Anthropic has done more, on balance, to drive a race to the bottom. (Disclosure: Future Perfect is funded in part by the BEMC Foundation, whose major funder was also an early investor in Anthropic; they don’t have any editorial input into our content.)
In fact, part of the problem may be that Anthropic continues to treat this as a classic arms race. Specifically, the company seems to model it as a prisoner’s dilemma: If Anthropic stopped building more powerful AI, another player — whether it’s China, OpenAI, or another company — would just build it anyway. In this kind of dilemma, if the other player is going to defect, it’s rational for you to defect, even though nobody actually wants that outcome.
But what if this framing misses the point?
Game theory can make a huge gamble seem sophisticated, while making the obvious alternative — just don’t build a doom machine — seem naive. Yet the fact that Anthropic and others are now racing toward RSI makes it much more likely that we could lose control of AI and start to see massive damage to our world. When the cost of defecting from safety is that you create a serious risk of catastrophic harm — or of all humanity, including you, going extinct — is that still a “rational” move?
Instead of telling yourself that you can galaxy-brain your way through this with fancy enough game theory, maybe you’re better off just thinking of this as something much simpler: Russian roulette.
More and more leading voices in AI are starting to recognize that.
“We should stop AI progress now, starting unilaterally with Anthropic and/or OpenAI stopping — and then broadening out as a second step to more AI companies and negotiations with China,” Geoffrey Irving, who was until recently the chief scientist at the UK AI Security Institute, told me. “Unilateral pauses can’t hold for long, but they are the right first step: When you’re doing a very dangerous thing, the correct action is stop, even if others are also doing dangerous things.”
Anthropic did not respond to a request for comment.
Let’s be clear: The question is not whether Anthropic has done anything good on the safety front. It has. Standing up to the Pentagon, temporarily holding back access to frontier models, and publicly advocating for safer AI policies are a few examples, among others.
But the real question is whether the company’s “race to the top” strategy is working on balance — whether it’s net-negative or net-positive for safety.
Nathan Calvin, the chief strategy officer at AI safety advocacy organization Encode, recently set off a firestorm among tech watchers, writing on X, “When I look at many notable actions that Anthropic has taken over the past year, that evidence seems most consistent to me with the view that Anthropic is primarily trying to win the race to superintelligence, and that other considerations (including safety) are fundamentally secondary.” Here’s his list of what Anthropic has done:
And this is all just in the past year. If we rewind the tape further back, we see Anthropic pushing back on a landmark bill to regulate AI. Rewind even further back, and we see indications that Anthropic’s core strategy may have been flawed from the start. According to Reuters, in late 2022, OpenAI CEO Sam Altman heard rumors that Anthropic was building an AI-powered chatbot. So he ordered his employees to ship an OpenAI chatbot first. That chatbot was called ChatGPT. We all know the rest.
The rivalry between OpenAI and Anthropic seems not only to have hastened the arrival of generative AI, but also to keep pushing release dates forward to this day. The two companies have been known to release new or cheaper models within hours of each other, each one keen to protect its market share.
Even if you think Anthropic’s behavior was justified in 2022 because publicly demonstrating the risks of AI also demonstrates the need to regulate AI — a common claim within the company and AI discourse writ large — you have to ask: Is it still justifiable now, when AI has reached this level of power and sophistication?
Anthropic has indicated that it’s getting closer and closer to achieving RSI. In January, Amodei himself wrote, “I am concerned that LLMs are approaching (or may already have reached) the knowledge needed to create and release [biological weapons] end-to-end, and that their potential for destruction is very high. Some biological agents could cause millions of deaths.” And he wrote in September, “Given the accelerating rate of AI capability development, it’s my worry that in 6-12 months such a swarm could be capable of taking over the entire internet with a persistent botnet.”
But at this point, Amodei and his staffers are themselves perhaps the primary ones accelerating the rate of AI capability development.
“Anthropic seems to be doing the whole RSI thing more than anyone else, other than perhaps OpenAI,” Calvin told me. “And I just think that seems insane given their publicly stated views on the risks.”
“I think people at Anthropic should reckon more with the fact that it seems like the modal outcome is that the largest impact that Anthropic will have had is dramatically accelerating the race to RSI,” he continued. “On the current trajectory, everything else they will have done and thought and ideals they had will be a rounding error compared to their contribution there.”
Anthropic should be willing to invite scrutiny on this point — and, if it believes it’s actually making the field safer than it otherwise would be, to show us the receipts. How? It could put before Congress a bill that actually strengthens safety, instead of weakening it. It could stick to its former Responsible Scaling Policy and get other companies to do the same, instead of jettisoning its commitments. It could influence other labs not to build RSI, instead of leading the charge.
In the absence of such evidence, the “race to the top” logic just doesn’t hold water.
Amodei has begun advocating for “pacing the frontier” of AI development. “We must slow the pace at which we improve the capabilities of AI models,” he wrote in September.
The catch is that Amodei’s plan to slow down is contingent on others agreeing to slow down too — other AI labs, to start with, and also other countries. But more experts are now saying that it would be better if a frontier company like Anthropic unilaterally stopped.
“I’m glad that he proposed that we pace, but like, now we have to actually do it — and it seems like he’s saying he’s not going to do it unilaterally,” Daniel Kokotajlo, a former researcher at OpenAI who became a prominent whistleblower, told me. “That’s not very helpful, Dario! You should lead. What’s all this about ‘racing to the top’ and ‘leading by example’ if you’re just waiting?”
Voluntary unilateral action by individual companies is of course no substitute for federal regulation and international treaties. But Irving argues that it’s the right first step, because if the companies wait for governments to rein them in, it may well be too late. Governments are notoriously slow. An individual company can act today.
There’s a perennial objection here: “But China!”
Yet the idea that we’re in a classic arms race with China has always been flawed.
For starters, AI is not just one thing with one purpose, like the atomic bomb. It’s a much more general-purpose technology, like electricity. So what matters isn’t simply who gets it first, but who uses it in which ways. Besides, if the harms from AI are massive, then those harms are relevant for the other party, too. The Chinese don’t want to see catastrophic harms to all of human civilization, either. So we shouldn’t write off the possibility of coordination.
Former Vox reporter Kelsey Piper recently wrote about a colloquium at Anthropic where one participant objected: A decade ago, Facebook considered acquiring the tech that would later power TikTok. But, facing pressure due to privacy and safety worries, the company passed. That didn’t end up protecting us from the scourges of a world with TikTok — all it did was hand China an early monopoly on the tech. Piper writes:
Was this a policy success? Did we want to repeat it?
We were obliged to grudgingly concede that it didn’t sound like a policy success. We did not want to repeat it. China might be no excuse for the U.S. labs to do reckless things, but it is a real consideration.
Our next step might be bilateral talks with China, then. But by the point where you agree that Anthropic’s plans might depend on bilateral talks with China, you’ve already conceded most of the steps of Anthropic’s worldview that seemed so much like madness 10 minutes ago: continue working on AI for now, try to negotiate a slowdown, but don’t abandon work at the frontier and all the advantages that come with it.
But TikTok is not a good analogy for RSI. The former may be a cesspool that rots our brains and amplifies ethnic hatred, but its dangers don’t compare to the dangers of RSI. The TikTok case may indeed be best approached as a classic prisoner’s dilemma. But with RSI, the potential downside is so catastrophic that anyone who proposes to save the world precisely by building the tech that endangers it is making a hell of a gamble on behalf of everyone else. At the very least, the burden of proof should be on them to prove their confidence is justified.
AI insiders often think those outside of their industry can’t possibly understand the technical, geopolitical, and moral complexities they’re facing. But commonsense morality is commonsense for a reason: It tracks intuitions that are usually right (such as, if you don’t want a doom machine, maybe don’t build a doom machine) and trains us to be appropriately skeptical of huge, counterintuitive bets to the contrary.
Precisely because the AI landscape is a complex system — there are so many actors (including different AI companies and different governments), and actions from each can trigger actions from others in ways that are hard to predict — believing you’re smart enough to take highly counterintuitive actions and end up saving the world as a result is an extremely risky gambit. Instead, maybe it’s best to just do the straightforwardly right thing: Stop.
To be clear, “stopping” doesn’t have to mean peeling back everything that’s been done in AI so far. The powerful AI systems that are already out there can’t be put back in the box. And we may need some AI to defend from the AI that’s already out there.
“I’m not saying shut down all of your data centers,” Calvin said. “But I’m saying that in particular, pushing the frontier on RSI and making it so that you’re going to be the company that is doing more fully autonomous development than anyone else in the world, does that actually make any fucking sense?”
Increasingly, both AI insiders and the public are shouting no.
Some insiders, perhaps aware that humanity is perfectly capable of shutting down overly dangerous tech development (we’ve done it before), are speaking out and quitting jobs at AI companies. At public gatherings, like the recent Pro-Human Assembly in Washington, DC, and the People’s AI Assembly in New York City, people are piling on the pressure to rethink the AI race. States are drawing up their own legislation to rein in the tech. And a single X post from Calvin got so much attention that no less than Amodei himself felt compelled to respond. The AI companies are being forced to pay attention.
Attention may not be all you need, but it’s a start.
© Ronen Zvulun/Reuters
© Athit Perawongmetha/Reuters
© Matt Chase
© Jason Henry for The New York Times
Login