We’re living through an AI-fueled “Black October,” and things are changing fast. Hackers are using AI to breach countless companies. AI tools have uncovered security flaws that could leave 70 percent of all websites vulnerable. Governments are scrambling to secure their sites. Banks, too. And because cyberhacks can hurt supply chains, some longtime AI watchers are going full prepper, stocking up on food, cash, and more.

In the center of this story is Anthropic, one of the world’s leading AI companies, which was explicitly founded on the promise that it would be a “good guy” company in a world of dubious tech giants.

Anthropic says all the latest developments are very serious. Its security chief wrote last week that things are “going to get a lot worse before they get better.” Two weeks ago the company warned that AI models could soon self-replicate on the internet-connected devices in your home, like smart fridges and robot vacuums. And its co-founder now says “the singularity” — when AI systems speed up technical progress in a way that’s hard for humans to comprehend — “is here.”

You might think, given its view of the risks, that Anthropic would be pumping the brakes on its cutting-edge AI progress. Just the opposite. Anthropic filed for a blockbuster initial public offering on Monday, chasing a valuation of $2 trillion even as it warns in the IPO filing that AI could “pose catastrophic or existential risks to humanity.” Every day, it is racing toward recursive self-improvement (RSI), where AI models are smart enough to build the next, even smarter AI models by themselves. 

How does Anthropic justify that?

As its CEO, Dario Amodei, reminded the world last week, he built the company on one big bet: that instead of driving a race to the bottom, where companies skimp on safety in their rush to beat competitors in the market, Anthropic could actually catalyze a “race to the top.” 

The idea is that if Anthropic proves it can build AI that’s both powerful and safe, the market will expect both — so other AI companies will then feel compelled to compete on safety, too. That may sound nice in theory, but it means Anthropic has to keep building ever-more-powerful AIs.

“The forces that drive the race to the top are hugely more effective if the company practicing them is at the frontier,” Amodei wrote. “And looking at that strategy as a whole, I’m proud of our track record and stand by it.”

In other words, Amodei is sticking to his bet. 

But Amodei is gambling with, by his and his company’s own telling, the future of humanity. If AI goes wrong, it could lead to our gradual disempowerment, a galling concentration of power, or even complete extinction.

So there’s a very basic question that the public deserves a good answer to right at this pivotal moment: What is the evidence that Anthropic’s stated strategy — compete at the frontier so as to drive a “race to the top” — is actually successful overall?

From the outside, it looks like Anthropic has done more, on balance, to drive a race to the bottom. (Disclosure: Future Perfect is funded in part by the BEMC Foundation, whose major funder was also an early investor in Anthropic; they don’t have any editorial input into our content.)

In fact, part of the problem may be that Anthropic continues to treat this as a classic arms race. Specifically, the company seems to model it as a prisoner’s dilemma: If Anthropic stopped building more powerful AI, another player — whether it’s China, OpenAI, or another company — would just build it anyway. In this kind of dilemma, if the other player is going to defect, it’s rational for you to defect, even though nobody actually wants that outcome.

But what if this framing misses the point? 

Game theory can make a huge gamble seem sophisticated, while making the obvious alternative — just don’t build a doom machine — seem naive. Yet the fact that Anthropic and others are now racing toward RSI makes it much more likely that we could lose control of AI and start to see massive damage to our world. When the cost of defecting from safety is that you create a serious risk of catastrophic harm — or of all humanity, including you, going extinct — is that still a “rational” move? 

Instead of telling yourself that you can galaxy-brain your way through this with fancy enough game theory, maybe you’re better off just thinking of this as something much simpler: Russian roulette.

More and more leading voices in AI are starting to recognize that. 

“We should stop AI progress now, starting unilaterally with Anthropic and/or OpenAI stopping — and then broadening out as a second step to more AI companies and negotiations with China,” Geoffrey Irving, who was until recently the chief scientist at the UK AI Security Institute, told me. “Unilateral pauses can’t hold for long, but they are the right first step: When you’re doing a very dangerous thing, the correct action is stop, even if others are also doing dangerous things.”

Anthropic did not respond to a request for comment. 

The case that Anthropic has done more harm than good

Let’s be clear: The question is not whether Anthropic has done anything good on the safety front. It has. Standing up to the Pentagon, temporarily holding back access to frontier models, and publicly advocating for safer AI policies are a few examples, among others.

But the real question is whether the company’s “race to the top” strategy is working on balance — whether it’s net-negative or net-positive for safety. 

Nathan Calvin, the chief strategy officer at AI safety advocacy organization Encode, recently set off a firestorm among tech watchers, writing on X, “When I look at many notable actions that Anthropic has taken over the past year, that evidence seems most consistent to me with the view that Anthropic is primarily trying to win the race to superintelligence, and that other considerations (including safety) are fundamentally secondary.” Here’s his list of what Anthropic has done:

And this is all just in the past year. If we rewind the tape further back, we see Anthropic pushing back on a landmark bill to regulate AI. Rewind even further back, and we see indications that Anthropic’s core strategy may have been flawed from the start. According to Reuters, in late 2022, OpenAI CEO Sam Altman heard rumors that Anthropic was building an AI-powered chatbot. So he ordered his employees to ship an OpenAI chatbot first. That chatbot was called ChatGPT. We all know the rest.

The rivalry between OpenAI and Anthropic seems not only to have hastened the arrival of generative AI, but also to keep pushing release dates forward to this day. The two companies have been known to release new or cheaper models within hours of each other, each one keen to protect its market share. 

Even if you think Anthropic’s behavior was justified in 2022 because publicly demonstrating the risks of AI also demonstrates the need to regulate AI — a common claim within the company and AI discourse writ large — you have to ask: Is it still justifiable now, when AI has reached this level of power and sophistication?

Anthropic has indicated that it’s getting closer and closer to achieving RSI. In January, Amodei himself wrote, “I am concerned that LLMs are approaching (or may already have reached) the knowledge needed to create and release [biological weapons] end-to-end, and that their potential for destruction is very high. Some biological agents could cause millions of deaths.” And he wrote in September, “Given the accelerating rate of AI capability development, it’s my worry that in 6-12 months such a swarm could be capable of taking over the entire internet with a persistent botnet.”

But at this point, Amodei and his staffers are themselves perhaps the primary ones accelerating the rate of AI capability development.

“Anthropic seems to be doing the whole RSI thing more than anyone else, other than perhaps OpenAI,” Calvin told me. “And I just think that seems insane given their publicly stated views on the risks.” 

“I think people at Anthropic should reckon more with the fact that it seems like the modal outcome is that the largest impact that Anthropic will have had is dramatically accelerating the race to RSI,” he continued. “On the current trajectory, everything else they will have done and thought and ideals they had will be a rounding error compared to their contribution there.”

Anthropic should be willing to invite scrutiny on this point — and, if it believes it’s actually making the field safer than it otherwise would be, to show us the receipts. How? It could put before Congress a bill that actually strengthens safety, instead of weakening it. It could stick to its former Responsible Scaling Policy and get other companies to do the same, instead of jettisoning its commitments. It could influence other labs not to build RSI, instead of leading the charge.

In the absence of such evidence, the “race to the top” logic just doesn’t hold water.

What if Anthropic just…stopped?

Amodei has begun advocating for “pacing the frontier” of AI development. “We must slow the pace at which we improve the capabilities of AI models,” he wrote in September.

The catch is that Amodei’s plan to slow down is contingent on others agreeing to slow down too — other AI labs, to start with, and also other countries. But more experts are now saying that it would be better if a frontier company like Anthropic unilaterally stopped.

“I’m glad that he proposed that we pace, but like, now we have to actually do it — and it seems like he’s saying he’s not going to do it unilaterally,” Daniel Kokotajlo, a former researcher at OpenAI who became a prominent whistleblower, told me. “That’s not very helpful, Dario! You should lead. What’s all this about ‘racing to the top’ and ‘leading by example’ if you’re just waiting?”

Voluntary unilateral action by individual companies is of course no substitute for federal regulation and international treaties. But Irving argues that it’s the right first step, because if the companies wait for governments to rein them in, it may well be too late. Governments are notoriously slow. An individual company can act today.

There’s a perennial objection here: “But China!”

Yet the idea that we’re in a classic arms race with China has always been flawed.

For starters, AI is not just one thing with one purpose, like the atomic bomb. It’s a much more general-purpose technology, like electricity. So what matters isn’t simply who gets it first, but who uses it in which ways. Besides, if the harms from AI are massive, then those harms are relevant for the other party, too. The Chinese don’t want to see catastrophic harms to all of human civilization, either. So we shouldn’t write off the possibility of coordination.

Former Vox reporter Kelsey Piper recently wrote about a colloquium at Anthropic where one participant objected: A decade ago, Facebook considered acquiring the tech that would later power TikTok. But, facing pressure due to privacy and safety worries, the company passed. That didn’t end up protecting us from the scourges of a world with TikTok — all it did was hand China an early monopoly on the tech. Piper writes:

Was this a policy success? Did we want to repeat it?

We were obliged to grudgingly concede that it didn’t sound like a policy success. We did not want to repeat it. China might be no excuse for the U.S. labs to do reckless things, but it is a real consideration.

Our next step might be bilateral talks with China, then. But by the point where you agree that Anthropic’s plans might depend on bilateral talks with China, you’ve already conceded most of the steps of Anthropic’s worldview that seemed so much like madness 10 minutes ago: continue working on AI for now, try to negotiate a slowdown, but don’t abandon work at the frontier and all the advantages that come with it.

But TikTok is not a good analogy for RSI. The former may be a cesspool that rots our brains and amplifies ethnic hatred, but its dangers don’t compare to the dangers of RSI. The TikTok case may indeed be best approached as a classic prisoner’s dilemma. But with RSI, the potential downside is so catastrophic that anyone who proposes to save the world precisely by building the tech that endangers it is making a hell of a gamble on behalf of everyone else. At the very least, the burden of proof should be on them to prove their confidence is justified.

AI insiders often think those outside of their industry can’t possibly understand the technical, geopolitical, and moral complexities they’re facing. But commonsense morality is commonsense for a reason: It tracks intuitions that are usually right (such as, if you don’t want a doom machine, maybe don’t build a doom machine) and trains us to be appropriately skeptical of huge, counterintuitive bets to the contrary. 

Precisely because the AI landscape is a complex system — there are so many actors (including different AI companies and different governments), and actions from each can trigger actions from others in ways that are hard to predict — believing you’re smart enough to take highly counterintuitive actions and end up saving the world as a result is an extremely risky gambit. Instead, maybe it’s best to just do the straightforwardly right thing: Stop.   

To be clear, “stopping” doesn’t have to mean peeling back everything that’s been done in AI so far. The powerful AI systems that are already out there can’t be put back in the box. And we may need some AI to defend from the AI that’s already out there.

“I’m not saying shut down all of your data centers,” Calvin said. “But I’m saying that in particular, pushing the frontier on RSI and making it so that you’re going to be the company that is doing more fully autonomous development than anyone else in the world, does that actually make any fucking sense?”         

Increasingly, both AI insiders and the public are shouting no. 

Some insiders, perhaps aware that humanity is perfectly capable of shutting down overly dangerous tech development (we’ve done it before), are speaking out and quitting jobs at AI companies. At public gatherings, like the recent Pro-Human Assembly in Washington, DC, and the People’s AI Assembly in New York City, people are piling on the pressure to rethink the AI race. States are drawing up their own legislation to rein in the tech. And a single X post from Calvin got so much attention that no less than Amodei himself felt compelled to respond. The AI companies are being forced to pay attention. 

Attention may not be all you need, but it’s a start.

Editor’s note, October 18, 8 am ET: We’re bringing you some of our best-loved Your Mileage May Vary columns while Sigal Samuel is on parental leave. The one below was originally published in September 2024.

This unconventional advice column offers you a unique framework for thinking through moral dilemmas. It’s based on value pluralism: the idea that each of us has multiple values that are equally valid but that often conflict with each other. Submit your own question here.

---

I think I have a duty to help people much poorer than me, and I give 10 percent of my salary to charities that I think are effective at preventing early death due to poverty. I also live in a city with a lot of visible homelessness, and am often solicited for money. My brain says that this is not an effective way to help people; the people asking might not be the neediest among the homeless in my city, and the people I’m sending malaria bednets and pills to are even needier. At the same time, I feel callous simply ignoring all these requests. What should I do?

Dear Would-Be Optimizer,

Nine times out of ten, when someone’s got an ethical dilemma, I think it’s because a couple of their core values are conflicting with each other. But you’re that tenth case. I say that because I don’t actually believe your question represents a battle royale between two different values. I think there’s one core value here — helping people — and one strategy that’s masquerading as a value.

That strategy is optimization. I can tell from your phrasing that you’re really into it. You don’t just want to help people — you want to help people as effectively as possible. Since extreme poverty is concentrated in developing countries, and since your dollar goes much further there than it would in your home country, your optimizing impulse is telling you to send your charity money abroad. 

Optimization started as a technique for solving certain math problems, but our society has elevated it to the status of a value — arguably one of the dominant values in the Western world. It’s been on the rise since the 1700s, when utilitarian thinkers seeded the idea that both economics and ethics should focus on maximizing utility (meaning, happiness or satisfaction): Just calculate how much utility each action would produce, and choose the one that produces the most.

You can see this logic everywhere in modern life — from work culture, with its emphasis on productivity hacks and agile workflows, to wellness culture, with its emphasis on achieving perfect health and optimal sleep. The mandate to “Live your best life!” is turbocharged by Silicon Valley, which urges us to quantify every aspect of ourselves with Fitbits, Apple Watches, and Oura Rings, because the more data you have on your body’s mechanical functions, the more you can optimize the machine that is you. 

Optimization definitely has its place, including in the world of charity. Some charitable organizations are much more effective than others trying to achieve the same goal. All things being equal, we don’t want to blow all our money on the wildly ineffective ones. Effective altruists, members of the utilitarian-flavored social movement that aims to do the most good possible, are fond of noting that the most effective charities out there actually produce 100 times more benefit than the average ones. Why not get the biggest bang for your buck? 

The problem is that we’ve stretched optimization beyond its optimal limits. We try to apply it to everything. But not every domain in life can be optimized, at least not without compromising on some of our values. 

In your case, you’re trying to optimize how much you help others, and you believe that means focusing on the neediest. But “neediest” according to what definition of needy? You could assume that financial need is the only type that counts, so you should focus first on lifting everyone out of extreme poverty, and only then help people in less dire straits. But are you sure that only the brute poverty level matters?

Consider an insight from the Jewish tradition. The ancient rabbis were exquisitely sensitive to the psychological needs of poor people, and they argued that these needs should also be taken into account. So they decreed that you shouldn’t only give poor people enough money to survive on — they need to have more than that so they themselves can give charity to others. As Rabbi Jonathan Sacks writes, “On the face of it, the rule is absurd. Why give X enough money so that he can give to Y? Giving to Y directly is more logical and efficient. What the rabbis understood, however, is that giving is an essential part of human dignity.” 

The rabbis also figured that those who used to be well-off but who fell into poverty might feel an especially acute sense of shame. So they suggested helping these people save face by offering them not just bare necessities, but also — when possible — some of the nicer things that graced their former lifestyles. In the Talmud, we hear about one rabbi who gave a newly poor person a fancy meal, and another who acted as the person’s servant for a day! Clearly, the ancient rabbis weren’t only aiming to alleviate poverty. They were also alleviating the shame that can accompany it.

The point is that there are many ways to help people and, because they’re so different, they don’t submit to direct comparison. Comparing poverty and shame is comparing apples to oranges; one can be measured in dollars, but the other can’t. Likewise, how can you ever hope to compare preventing malaria with alleviating depression? Saving lives versus improving them? Or saving the life of a kid versus saving the life of an adult? 

Yet if you want to optimize, you need to be able to run an apples-to-apples comparison — to calculate how much good different things do in a single currency, so you can pick the best option. But because helping people isn’t reducible to one thing — it’s lots of incommensurable things, and how to rank them depends on each person’s subjective philosophical assumptions — trying to optimize in this domain will mean you have to artificially simplify the problem. You have to pretend there’s no such thing as oranges, only apples.

And when you try to do that, an unfortunate thing happens. You end up rushing past all the unhoused people in your city and, as you put it, you “feel callous simply ignoring all these requests.” Ignoring these human beings comes at a cost, not only to them, but to you. It has a damaging effect on your moral conscience, which feels moved to help but is being told not to.

Even some leaders in effective altruism and the adjacent rationalist community recognize this as a problem and advise people not to shut up that part of themselves. Rationalist Eliezer Yudkowsky, for example, says it’s okay to donate some money to causes that make us feel warm and fuzzy but that aren’t producing maximum utility. His advice is to “purchase fuzzies and utilons separately” — meaning, devote one pot of money to pet causes and another (much bigger) pot of money to the most cost-effective charities. You can, he says, get your warm fuzzies by volunteering at a soup kitchen and “let that be validated by your other efforts to purchase utilons.” 

I would also suggest diversifying your giving portfolio, but it’s not because I think you need to “validate” the warm fuzzies. Instead, it’s because of another value: integrity. 

When 20th-century British philosopher and critic of utilitarianism Bernard Williams talked about integrity, he meant it in the literal sense of the word, which has to do with a person’s wholeness (think of related words like “integration”). He argued that moral agency does not sit in a contextless vacuum — it is always some specific person’s agency, and as specific people we have specific commitments. 

For example, a mother has a commitment to ensuring her kid’s well-being, over and above her general wish for all kids everywhere to be well. Utilitarianism says she has to consider everyone’s well-being equally, with no special treatment for her own kid — but Williams says that’s an absurd demand. It alienates her from a core part of herself, ripping her into pieces, wrecking her wholeness — her integrity. 

It sounds like that’s what you’re feeling when you pass a person experiencing homelessness and ignore them. Ignoring them makes you feel bad because it alienates you from the part of you that is moved by this person’s suffering — that sees the orange but is being told there are only apples. That core part of you is no less valuable than the optimizing part, which you liken to your “brain.” It’s not dumber or more irrational. It’s the part that cares deeply about helping people, and without it, the optimizing part would have nothing to optimize! 

So rather than trying to override it, I would encourage you to honor your wish to help in all its fullness. You won’t be able to run a direct apples-to-apples comparison, but that’s okay. Different types of help are useful in their own way and you can divvy up your budget between them, even though there’s no perfect formula to spit out the “optimal” allocations. 

Diversifying your giving portfolio might look something like this. You keep a small amount of cash or gift cards on you, which you hand out to unhoused people you encounter directly. You put aside a larger amount to donate to a local or national charity with a strong track record. And you devote another amount to a highly effective charity abroad. 

You might feel annoyed that there’s no universal mathematical formula that can tell you the best thing to do. If so, I get it. I want the magic formula too! But I know that desire is distinct from the core value here. Don’t let optimization eat the real value you hold dear. 

Bonus: What I’m reading

• I recently read Optimal Illusions, a book by mathematician Coco Krumme that traces the roots of optimization’s overreach. As she puts it, “Over the past century, optimization has made an impressive epistemic land grab.” 

• When torn between competing moral theories, does it make sense to diversify your donations in proportion to how much you believe in each theory? Some philosophers argue against that view, but Michael Plant and coauthors defend it in this new paper.

• This gorgeously written essay by anthropologist Manvir Singh introduced me to the term “cooperating without looking” (or, because it’s a New Yorker essay, “coöperating without looking”). This “tendency to willfully ignore costs and benefits when helping others” — to help without calculating what you’ll gain from the altruistic act — is “a key feature of both romantic love and principled behavior.” When we help this way, people trust us more. 

Editor’s note, October 4, 8 am ET: We’re bringing you some of our best-loved Your Mileage May Vary columns while Sigal Samuel is on parental leave. The one below was originally published last year.

This unconventional advice column offers you a unique framework for thinking through moral dilemmas. It’s based on value pluralism: the idea that each of us has multiple values that are equally valid but that often conflict with each other. Submit your own question here.

---

My partner and I did IVF recently and luckily we ended up with several embryos. One of those will be implanted in me so I hopefully get pregnant. But which one?

My fertility clinic is encouraging me to do PGT-A testing, which would screen for things like Down syndrome and would also let me see the sex of each embryo so I could potentially choose whichever I want. (And to be honest, I really want a girl!) I’m also hearing about these new companies that can test embryos for all kinds of things — like how smart the kid might be, how tall they’ll get, and whether they’ll have a lower chance of getting breast cancer, diabetes, mental illness, stuff like that.

I don’t know how to feel about any of this. On the one hand, it feels weirdly eugenics-y to do so much tinkering with embryos, and I can’t help feeling a little pressured into overmedicalizing everything with these expensive tests. But then again, if I could do something to make my baby healthier and happier, don’t I kind of have to do everything in my power? What does a parent owe their child?

Dear Expecting,

This is such an exciting — and also confusing — time. You’re about to have your first kid, and that comes with all sorts of fears and insecurities and hopes and dreams. You want to do the right thing — no, the best thing! — for your child. It’s a beautiful impulse. Unfortunately, it makes it easy for companies to prey on you.

So let me walk you through what these companies can and can’t really provide. After we talk about the science, we can talk about the ethics. 

The simpler test you mentioned, PGT-A, has been around since the 1990s. It can indeed screen for conditions like Down syndrome and let you see the sex of each embryo. Mind you, that doesn’t predict what gender your child will ultimately end up identifying with — more on that soon.

The newer testing you mentioned is called polygenic testing. The conditions it screens for are complex ones, influenced by thousands of genes. We’re talking about everything from breast cancer to depression. Some companies even offer screenings not for disease prevention, but for genetic enhancement — so you can select for a taller kid or a kid with higher IQ.

Polygenic testing companies claim to predict each embryo’s risk for each condition, and the potential reduction of risk you’ll get if you choose one embryo over another. As the statistical geneticist Sasha Gusev explains, it’s very, very easy to be misled by this claim. If the company claims to lower absolute risk of Type 2 diabetes by 12 percent, you might think this means, “The chance of my kid ever developing diabetes just went down by 12 percent — great!” But really what it means is that, all else equal, a kid who would’ve developed diabetes is likely now at the high end of pre-diabetes instead, or just develops diabetes a few years later. The same holds true for other conditions, like breast cancer. Sure, if your kid is going to develop diabetes or breast cancer, you’d rather they develop it later — but it’s not the bill of perfect health customers might imagine.  

“Typical risk reduction estimates tend to exploit statistical assumptions to overstate their benefits and lead to confusion,” Gusev writes.

It’s also important to note that there’s a difference between a physical disease like cancer, which is straightforwardly and universally negative, and a complex mental health condition like depression. For one thing, polygenic scores don’t perform as well in psychiatry, as the psychologist Eric Turkheimer notes. Plus, most mental health conditions come in mild to moderate to severe forms, and it’s not clear that we should want to get rid of mild or moderate manifestations. 

“Sure, it would be nice if no one ever got depressed, but what would that actually mean?” Turkheimer writes. “People are going to be sad, and the most modern models of psychiatric disorders view most of them as extremes on continuous dimensions of normal behavior. I don’t think we want some kind of genetic version of Soma.”

To give a personal example: I have generalized anxiety with some obsessive tendencies. Is that a major pain in the butt sometimes? You bet. But if I could wave a magic wand and get rid of it, would I? No. In my experience, this trait is linked to another trait I deeply value: my creativity. The same thing that propels me down rabbit holes of worry also propels me down rabbit holes of getting super curious about some topic, researching it into the wee hours of the morning, and then writing a novel or children’s book about it.

That brings me to one other scientific issue: pleiotropy. It’s a fancy term for a simple truth, which is that genes can have an effect on more than one trait. When it comes to behavioral polygenic testing, we actually know very little about which genes cause what — a gene that’s correlated with depression or anxiety may also be correlated with a bunch of other things. So, if everyone were to start selecting embryos to try to protect their kids from depression or anxiety, what would happen? We don’t know! 

Pleiotropy is a scientific issue, but it’s also a moral issue, because in trying to protect your kid from a negative outcome, you could also be accidentally barring them from a positive one.

And that’s just the tip of the moral iceberg.

Another issue is the potential psychological harm that can result from commodifying our kids — treating them like consumer objects with selectable parts. Let’s say you use polygenic testing to try to select for high IQ. Even if the company’s predictor is right 80 percent of the time, that means it’s wrong 20 percent of the time. What happens if it’s wrong about your kid, and they end up with a low IQ instead of a high one? How will your kid feel? Unless you hide the truth from them, they’ll know that they are not what you wanted, that they failed to live up to the vision you went out of your way to chase.

The same goes for sex selection via good old PGT-A. I know you really want a girl. But what would happen if you select for a girl, and then your child grows up and turns out to be trans? They may feel an extra sense that they’re letting you down by not conforming to the gender expectations you had for them.

Then there are the ethical questions around unequal access. Polygenic testing isn’t cheap. (One company, Orchid, charges $2,500 per screened embryo. Test 10 embryos and that’ll cost you $25,000.) So, at least in the short term, that raises the specter of entrenching a caste system, where the gap between rich and poor expands dramatically as the rich get smarter and stronger.

Technologies tend to get cheaper over time, so maybe everyone will ultimately be able to afford polygenic testing. But that raises another problem: implicit coercion. If more and more of us make ourselves smarter and stronger, we may create a society in which everyone feels pressure to alter their child’s biology — even if they don’t want to. To refuse embryo selection would mean to put a child at a huge professional disadvantage, or to face moral condemnation for making “suboptimal” kids when optimization is possible. Society may no longer offer health insurance benefits or accommodations for kids with disabilities (or even non-enhanced IQ), reasoning that “it was the parents’ choice.” 

Some philosophers, like Julian Savalescu, already argue that parents have a “moral obligation to create children with the best chance of the best life.” Polygenic testing companies love that argument; Orchid founder Noor Siddiqui tells parents that “the responsible decision” is to “protect their children” from disease, using language like, “What if you could have stopped it…but chose not to?” And we generally live in a culture that’s obsessed with optimization. 

So it’s no wonder that you find yourself asking, “If I could do something to make my baby healthier and happier, don’t I kind of have to do everything in my power?” 

It’s a view that will be very appealing to utilitarians, who believe the right action is the action that maximizes well-being or happiness. But maximizing principles can be dangerous. If we’re laser-focused on maximizing one value, however noble it may be, that can lead us to overlook a host of other important values. Think about diversity, for example; polygenic testing could lead to a less diverse world. 

And it’s not even clear how to maximize well-being given that each embryo will have some downsides: How would you choose between one embryo with a slightly elevated risk for schizophrenia, another with a moderately elevated risk for cancer, and a third with a high risk for Alzheimer’s? You’d have to decide how to rank the importance of each condition, which would require a dizzying bit of moral math: How much weight do you put on the longevity associated with each condition, how much on patient-reported quality of life for each condition, and how much on your own ability to cope with each — and society’s ability (which will change over the years)? Plus, even if the screening does end up optimizing life for your individual child, it could have negative effects on the population level, for the reasons we covered above.

As the Nobel laureate Herbert Simon pointed out, many problems contain way more variables and way too much uncertainty for maximizing along a single axis to be feasible or even desirable. So it often makes sense to just go with an option that’s “good enough.” He coined the term “satisficing” — a portmanteau of “satisfying” and “sufficing” — to describe opting for this good enough choice.

What would satisficing look like in the context of embryos? If there’s a very serious condition that runs in your family — like Tay-Sachs or the BRCA mutation that markedly increases cancer risk — then I do think there’s a strong case for screening for that. These conditions are debilitating and can lead to early death. Nobody wants them. Genetic testing that lets us prevent them is an awesome gift. 

Then there are conditions that can include suffering, like autism, but can also be very compatible with a happy life. Mental health conditions in particular are heavily socially constructed — a lot of the associated suffering comes about because society constructs a certain biological reality as a problem and doesn’t support the people who have it. In these cases, if you want to give your child a “good enough” or even “optimized” life, remember that you don’t have to achieve that by biological means; working to change the social environment can be just as important, and it doesn’t carry the societal risks associated with polygenic testing. 

Ultimately, when considering what counts as a “good enough” life, each parent will draw the line in a slightly different place. And there’s a part of that that can’t be boiled down to objective facts about disease, because this is also about how well-resourced the parent feels — emotionally, socially, and financially — to deal with a certain condition. What feels manageable to one parent may feel crushingly hard to another. 

So, I can’t tell you exactly which screenings to do and which to forgo. What I do want to tell you is to resist being bullied or shamed by others, especially those with a profit incentive to push you in a certain direction.

As the epidemiologist Abby Lippman has noted, genetic testing is supposed to offer parents more choice, but it too often has the opposite effect. When someone like Orchid’s Siddiqui says about a disease, “What if you could have stopped it…but chose not to?” — well, that language preys on parents’ worst fears and insecurities. It holds parents hostage while pretending to be giving them more autonomy. 

You are nobody’s hostage. You are a moral agent free to look at the many different relevant factors, weigh them all in the balance, and then make the choice that works well enough for your family.   

Bonus: What I’m reading

• Silicon Valley is obsessed with IQ, so it’s no surprise that tech execs there are using polygenic testing in hopes of creating supersmart babies. This article shows the kind of mental pretzels customers get into when trying to figure out whether to select the embryo with the highest predicted IQ or that with the lowest risk of disorders. One couple asks: “How much additional risk of ADHD cancels out against 10 extra IQ points?”

• Rereading “The Last Children of Down Syndrome,” Sarah Zhang’s 2020 Atlantic feature on prenatal testing, it struck me that when people think “prenatal testing,” Down syndrome is often the first condition that comes to mind, even though it’s not the most severe or straightforwardly negative (it’s compatible with a happy life). It’s just that Down syndrome was easy to test for, so we started testing for it early. I wonder if that built up the condition as a bogeyman in the expectant parent’s imagination — if the fact that we could screen for it shaped the public’s perception of Down syndrome as something that should be screened for and potentially screened out.

• One trait that many people assume we should select for is a great memory. But I highly recommend the short story “Funes the Memorious” by Jorge Luis Borges. It’s a reminder that having too good a memory is not always a benefit.  

Vox Members got to read this story first. Support independent journalism and get exclusive access to stories like this by becoming a Vox Member today.

Editor’s note, September 20, 8 am ET: We’re bringing you some of our best-loved Your Mileage May Vary columns while Sigal Samuel is on parental leave. The one below was originally published last September.

This unconventional advice column offers you a unique framework for thinking through moral dilemmas. It’s based on value pluralism: the idea that each of us has multiple values that are equally valid but that often conflict with each other. Submit your own question here.

---

The Jewish High Holidays are coming up, and that includes Yom Kippur. It’s a holiday that encourages people to reflect on their behavior and make amends. That’s all well and good, but I’m someone who struggles with scrupulosity — constantly worrying about my morality and if I’m doing The Most Possible Good™.

In practice this is far more paralyzing than motivating. Fixating on the ethical implications of all possible decisions makes it harder to take any action, and I’ve lost hours scouring my memories of my past behavior for immorality like a football player watching footage of their games to analyze what they could do differently. Guilt simply isn’t serving me, but I worry that saying to hell with all that means I’ll stop striving to be a better person and become morally complacent.

I’ve observed Yom Kippur for decades, and don’t want to simply avoid the day. But the holiday is a moral scrupulosity trigger. How do you think I should approach this? I want to stop feeling guilty for letting guilt get in my own way.

Dear Scrupulous, 

Have you ever heard the story about what happened when God decided to give the Bible to flesh-and-blood human beings? According to the ancient rabbis, the angels hated the idea. They argued that humans were deeply flawed mortals who didn’t deserve such a holy scripture; only angels could be worthy of it, so it should stay up in heaven. 

It fell to Moses to rebut the angels’ argument. He asked them: What do you angels need the Bible for? The Bible says not to murder, not to commit adultery, not to steal — do you have jealousy or other emotions that could lead you to do those things? The Bible says to honor your father and mother — but you don’t have parents, so how could you ever do that? And the Bible says to sanctify the Sabbath — but you never do any work, so how could you even honor the Sabbath by resting? 

The angels saw that Moses was right. Angels are really great at one thing: being perfect. But perfect creatures are static. They don’t experience painful challenges, they don’t grow, and they don’t make choices that add beauty to the world. We humans do those things. God gives the Bible to humans not to make them into angels — but to make them better at being the unique thing they are: human animals, with feelings, flaws, and all, that can learn to use their capabilities in more beautiful ways.

I think there’s a lot of wisdom in this story. And I want you to notice how far it is from demanding that humans do “The Most Possible Good™.” That language suggests a maximizing ethical theory like utilitarianism, which says that we have to do the action that produces the greatest good for the greatest number. In other words, we have to optimize. 

Moral optimization may be possible for angels, but not for humans. We each hold multiple values, and sometimes those values are in tension with each other, forcing us to strike a balance as best we can. We’re not omniscient beings who can know with certainty how best to strike the balance. 

What’s more, sometimes different kinds of moral good straight-up conflict with each other. Think of a woman who faces a trade-off: She wants to become a nun but also wants to become a mother. She can’t balance between those options — she has to choose. And what’s the better choice? We can’t say because the options are incommensurable. There’s no single yardstick by which to measure them so we can’t compare them to find out which is greater.

Given that this complexity is baked into the human condition, it’s impossible to be a perfect optimizing machine. And the more you try to force yourself to be that, the harder it’ll be for you to actually help others, because you’ll just be so burned out. As you’ve already discovered, the optimizing mindset is exhausting — you end up expending a lot of precious mental resources that you could be spending on something else. It can even lead to paralysis. And a lot of the time, there’s no knowably best decision.

So instead of trying to optimize everything, you can adopt a goal that’s humbler but more realistic: to live in line with your values as best you can. 

I know that can feel scary. Optimizing makes being human feel less risky. It provides a sense of control, and therefore a sense of safety. The unspoken premise is that if you optimize, you’ll never have to ask yourself: How could I screw up that badly? 

But there’s another way to feel safe. It’s about leaning into the fact that we are imperfect and vulnerable creatures and that even when we’re trying our hardest there will be some things that we do not do optimally.

Of course, we should still try to live in line with our values. But in those moments when we fall short, we shouldn’t berate ourselves, thinking, “I sinned!” In Hebrew, the word we typically translate as “to sin” (lachto) actually means “to miss the mark.” It’s the same word we’d use to describe someone with a bow and arrow who’s targeting the bullseye but misses it slightly. This is a useful image, because it reminds us just how normal it is to miss the mark. Just as the archer’s arrow is buffeted around by the wind, we’re buffeted around by all the physical and psychological conditions acting upon us — naturally we won’t always hit the bullseye! And when we do miss the mark, we deserve compassion.

I know what you’re thinking: What if adopting this mindset means you’ll become morally complacent and let yourself off the hook too easily? This is one of the most common objections to practicing self-compassion. But research shows it’s not well-founded. In fact, psychologists have found that more self-compassionate people are better able to acknowledge when they’ve made a mistake. They’re more likely to want to apologize and make amends to others when they mess up. And they try to do better the next time around. Why? Because, to them, mistakes don’t feel so psychologically damning. That allows them to take more — not less — responsibility for their actions.

Yom Kippur can feel terrifying when any mistake you’ve made over the past year seems damning. But according to the ancient rabbis, Yom Kippur is not meant to be a somber day — it’s one of the happiest days of the year! After all, it was on Yom Kippur that Moses descended from Mount Sinai carrying the second set of tablets inscribed with the Ten Commandments, ready to gift them to the people. 

You probably know what happened to the first set of tablets: Moses shattered them after seeing the Israelites engaged in idol worship. What’s less known is that, according to one rabbinic story, God’s response to the shattering of those tablets was to actually congratulate Moses. Why did God think breaking them was the right move? And what was different about the second set? 

While the first tablets had been fashioned by God and God alone, the second tablets were a human-divine collaboration: Moses carved the stone and God inscribed the words. And while the first tablets contained only the words of the Ten Commandments — a black-and-white, rule-based morality — the rabbis tell us that the second tablets contained within them all the stories and interpretations that Jewish sages would later develop.

In other words, God recognized that you can’t just give humans a list of moral rules and call it a day. Maybe that would work for angels, who live in a simplified, disembodied world, but our ethical life is just too messy and multifaceted to be captured by any single set of universally binding moral principles. Yet God chose people over angels anyway, inviting us into the collaboration and embracing our humanness rather than rejecting it.

So, to the rabbis, Yom Kippur was a happy day because they fully expected that God would accept and embrace messy humans. 

Please, don’t try to be more zealous than God.

When you’re taking action, by all means, aim your arrows as true as you can — try to hit the bullseye, the place that captures as much of what you value as possible. But once you’ve released the arrow from your bow, let it be. 

If it turns out that you missed the mark, that you acted suboptimally, I put it to you that that is okay. You are not an angel. You are not a perfect optimizing machine. You do not have access to a magical mathematical formula that can consider countless incommensurable variables and spit out the very best move with certainty. You are human and you do the best you can with what you’ve got. 

The wisdom of these millennia-old stories is that that’s good enough for God. Let it be good enough for you, too. 

Bonus: What I’m reading

• Writing this column reminded me of The Mindful Self-Compassion Workbook, written by the psychologists Kristin Neff and Chris Germer. It really helped me develop a self-compassion practice, which has in turn helped me get a grip on my own scrupulosity. I also strongly recommend the eight-week self-compassion course run out of Neff and Germer’s nonprofit, the Center for Mindful Self-Compassion.

• I’ve always associated the philosopher Thomas Nagel with questions about consciousness, but I recently learned that he’s also super interested in questions about religion. In a great essay called “Secular Philosophy and the Religious Temperament,” Nagel asks: What, if anything, does secular philosophy have to put in the place of religion? More specifically, can it answer the question: What is the underlying nature of the universe, and how can the human individual live in harmony with it? 

• In this Aeon essay, philosopher Elad Uzan argues that AI will not be able to solve ethics for us, despite what some people hope. Drawing on the mathematician Kurt Gödel’s famous incompleteness theorems, Uzan writes, “just as mathematics will always contain truths that lie beyond formal proof, morality will always contain complexities that defy algorithmic resolution.”