Most Americans hate their cable companies. But not the 200,000-odd residents of Chattanooga, Tennessee, a city nestled in the Appalachian foothills. 

For many people, the city might seem off the beaten path — enough so that when Spain’s national team chose the city as their base during the FIFA World Cup over the summer, the internet exploded with jeers of incredulity. And yet, as many an urban planner, and certainly many a local, will tell you, Chattanooga is a special place, known not just because the Glenn Miller Orchestra immortalized it as a scenic hub for the choo-choo train.

No, what really has put Chattanooga on the map recently is its ridiculously fast, municipally owned internet. 

Unlike almost everywhere else in the country, the fiber-optic cables that keep much of Chattanooga online were laid neither by Comcast nor Spectrum, but by the city itself. In a $220 million gamble, its public power utility, EPB, has helped transform the once struggling post-industrial city into an unlikely marvel of civic reinvention, and an inspiration for many other dozens of copycat towns around the country.  

When it debuted in 2010, at one gigabit per second, the city’s ultra-speedy broadband was among the fastest in the world, as fast as global meccas like Hong Kong, and over 200 times faster than the average top-tier offering across the US. “This makes Chattanooga, a mid-sized city in the South,” declared then-Mayor Ron Littlefield to the New York Times, “one of the leading cities in the world in its digital capabilities.” 

In the decade and a half since Chattanooga rebranded as “Gig City,” its trademark broadband has proved to be far more than a mere fancy gimmick. It has added billions of dollars to the city’s economy, lured in hundreds of new startups, and, according to EPB, become the single largest source of funding for local public schools, and delighted hundreds of thousands of local customers. 

If you’re not a tech company or a prolific streamer, of course, you may not need such intensive internet service. But the city’s broadband service is widely beloved locally for bringing 70 percent of households online at, by far, the lowest price per megabit of bandwidth in the area and quite possibly, among the lowest in the country. Consumer Reports perennially ranks EPB among America’s best providers, a sentiment apparently shared by the legions of Chattanoogans who take to social media to sing EPB’s praises every time it swoops in to fix an outage minutes after a storm hits, or bemoan their loss when they move away and witness the sludgy pace at — and high rates — which the rest of us browse and scroll. 

“Chattanooga turned out better than anyone could have hoped,” said Christopher Mitchell, director of the Community Broadband Networks Initiative, part of the Institute for Local Self-Reliance, a group that researches locally owned infrastructure, “and the one thing that hasn’t changed are their prices. It’s rare, remarkable even.” That original single-gig service runs $68 per month, and faster speeds aren’t too much more.  

But even as the benefits of Chattanooga’s internet supremacy have compounded, public broadband networks like it are still a rarity across the US, which you probably already know if you, like most Americans, have ever sensed your internet growing ever more lethargic, even as your fees climb month after month. 

There is a better way, and Chattanooga has long since shown the nation how. For better or worse, the internet drives much of our daily lives and the economy, and what makes the internet possible? Broadband, which is powered by those stiff plastic-sheathed cables filled with strands of thin glass or braids of copper wire, beaming pulses of light or electricity into your modem. And yet, the US has outsourced the provision of this precious public good almost entirely to de facto monopolies that keep our internet slower, more expensive, and above all, more infuriating than it needs to be. 

“There was a generation that built water systems and electric systems. There was a generation that built the roads,” Mitchell said. Today, however, “People are just afraid of the idea of the government being ambitious and doing what Chattanooga did.”

How Chattanooga built the world’s best broadband

Long before Chattanooga grew famous for its broadband, it was known, for a period of time, as “the dirtiest city of America,” a moniker bestowed by Walter Cronkite in 1969. 

According to local lore, the moon pie, which originated at a bakery in Chattanooga, came to be when a coal miner asked for a snack “as big as the moon” to sustain him for long shifts down the shaft. The coal he mined fueled the industrial boom that defined Chattanooga until the mid-20th century, when the city’s factories, furnaces, and railroads belched a smog so noxious, it melted women’s nylon stockings off their legs as they walked. 

By the 1980s, the city knew it needed to change, seeding a hunger for reinvention that experts told me would come to define its municipally-owned internet. By the 2000s, Chattanooga had more than cleaned up its air and its reputation, building a huge aquarium in the heart of downtown and carefully revitalizing its riverfront and natural areas. But in the early aughts, with the remaining heavy industry in deep retreat, and a fragile recovery threatened by the Great Recession, the city craved a new economic engine. 

The city’s EPB, which originated as a municipally owned electric utility in 1935, had already set out to build a smart electrical grid in 2008, retrofitting it with a two-way digital communications system that could detect and fix problems as they arise. This process would involve laying hundreds of miles of new fiber-optic cable lines along existing power lines, monitoring them and relaying information back and forth with operators. But as the EPB laid plans for its new grid-helping fiber, it realized that for roughly the same amount of effort and cost, it could repurpose those cables to also serve as a cutting-edge internet service. 

Laying those hundreds of miles of new cables, along every single power line in the city, is an expensive and time-consuming process. But since EPB already had power lines plugging into almost every home and business throughout the city, it didn’t even need permission to get it done. And once the cables are there, there’s no real limit on how much data they can carry, meaning that they’ll work just as well whether they’re relaying the power grid’s data exclusively or if they’re carrying petabytes upon petabytes of data from hundreds of thousands of internet customers. This same quality has made it easy for EPB to pump up the volume of data it offers over the years, eventually allowing it to become the first city in the nation to offer a 25-gigabit-per-second service, which it did for its customers in 2023. More recently, EPB became the first provider nationwide to hook its fiber-optic cables up to a quantum computer — which uses advanced physics to solve mind-bogglingly complex problems — and offer its use to researchers and customers across the city. 

“If we’re going to own that fiber communications network to help automate our grid, why not offer that as a service to our customers? We thought, ‘Why not go big? Why not be ambitious?’” Janet Rehberg, CEO of EPB, told me. “People thought we were crazy. And we said, ‘If we can do it, why not?’ So, we did.”

Equipped with a $111 million federal stimulus grant from the recession-era American Recovery and Reinvestment Act, plus about $222 million in municipal bonds, EPB was able to complete the project — initially expected to take a decade — in just two years. 

In 2010, Chattanooga became the first city in the nation to offer one-gigabit-per-second fiber optic internet. 

“This was a huge shift for our city’s fortunes,” Andy Berke, mayor of Chattanooga from 2013 to 2021, told me. “All of a sudden, we were a place where you could try to discover the next big thing.” 

Hundreds of startups flocked to the city for its super-fast broadband, with tech incubators setting up shop in shuttered storefronts and abandoned industrial factories dotted along the city’s downtown. In its first 10 years, EPB’s fiber network brought more than $5.3 billion in economic benefits and 10,420 jobs to the city, about 31 percent of all jobs created during that period, according to a study by the University of Tennessee at Chattanooga. This has transformed “the way we thought about ourselves and what we thought was possible technologically,” Berke said. 

The broadband investment has long since paid for itself and then some in the fees people pay to access its service. EPB expects to earn about $190 million over the next year alone from its telecommunications and enormously popular internet service, all of which will flow back into the business itself or into the city. As a result, EPB chose not to increase electricity rates at all this year, in stark contrast to much of the rest of the country. And the utility now pays tens of millions of dollars annually straight back into city services, like public schools. 

Under Berke’s mayoral administration, the city also integrated EPB into initiatives meant to get more Chattanoogans online. When the pandemic hit, and local schools went virtual, the city built a new program called EdConnect that saw EPB provide free internet to every family with a child who received free or reduced lunches, the first program of its kind nationwide, and one that continues to this day. 

“A lot of communities set up something similar during the pandemic, but most of those are done, they kind of faded away. But we made a commitment — our community did — for 10 years, and we’re already discussing what the next 10 years looks like,” said Sammy Lowdermilk, CEO of the Enterprise Center, the nonprofit that administers EdConnect. “We’ve heard from parents using online resources to look for a job, to create resumes, to do medical research. The entire family gets to benefit from it.”

Your city could have Chattanooga’s internet. It just has to try.

Over the years, Chattanooga’s publicly owned internet has drawn countless city planners and far-flung mayors to Tennessee’s fourth-largest city to learn just how they pulled it off.

But every time one visited Berke in his time in office, and every time he gave one of his “little sales pitches” on EPB’s success, over and over, they’d replied with the same refrain: “Oh, it’s incredible what you’ve done here. I mean, we can’t do it, but it’s great — good for you guys,” he recalled. Many cities don’t have a publicly owned electricity company at the ready as Chattanooga did, and many face significant barriers from state laws that restrict or heavily discourage the development of municipally owned broadband altogether. 

“The number one challenge for most infrastructure projects is political. It’s not financial. It’s not conceptual. It’s not technological. It’s the web of difficulties that make it harder to get it done.”

Andy Berke, former mayor of Chattanooga

Nevertheless, some places tried anyway, and many have succeeded — over 700 of them, according to Mitchell’s last count. In Utah, for example, a consortium of cities has spent the past few decades building out a municipally owned fiber network with the bold name Utopia. For many years, said Mitchell, the network suffered from severe financial and legal roadblocks. But even so, Utopia has managed to build out a system that Mitchell calls an “unqualified success” today, bringing tens of thousands of households online across the state. 

While only a tiny fraction of all broadband networks nationwide have some level of municipal ownership, strikingly, nine out of ten of the nation’s fastest broadband networks do or are part of a public-private partnership. 

However, Chattanooga’s brand of municipally owned broadband hasn’t spread nearly as widely as many people hoped it would, or think it could. And most of that, said Mitchell, can be attributed to the fact that large internet companies like Comcast or AT&T, clearly aware of how dominant well-run municipally owned networks can become, have spent the past several decades fighting to prevent more from coming online.

More than 12 states now have explicit laws barring or restricting cities from offering publicly owned broadband, which has made it difficult for many municipalities to create networks of their own. Even Chattanooga faced a brutal legal challenge when it tried to expand EPB’s coverage to rural and underserviced areas right outside of the city, the result of a 1999 state law meant to restrict municipally owned broadband access. 

“The number one challenge for most infrastructure projects is political. It’s not financial. It’s not conceptual. It’s not technological,” said Berke, but rather, it’s a matter of navigating the special interests and lobbyists that dominate the status quo. “It’s the web of difficulties that make it harder to get it done.” 

But that doesn’t mean that it’s impossible to pull off. Utopia, for example, has withstood a blitz of dark money attack ads likely bankrolled by cable companies in its effort to expand across the state. And in Colorado, hundreds of communities over the years across the state have voted to opt out of a 2005 law banning municipal broadband networks. This momentum ultimately proved strong enough to convince the state legislature to repeal the original ban in 2023.      

Even today, broadband remains out of reach in many parts of the country that are either too rural for for-profit networks to build there at all, or where it does exist, locals can’t afford it. Americans consistently pay more for slower service, compared with their counterparts in Europe or Asia. 

Combined, those barriers have left tens of millions of American households without internet access. With AI poised to further transform our economy, Chattanooga, it seems increasingly clear, could be among the first few cities outside of Silicon Valley to truly benefit from the biggest technological shift in generations, while many other post-industrial cities seem likely to fall even further behind because they don’t have the right digital infrastructure to thrive in the economy today.

Quietly chugging along on the banks of the Tennessee River, the city is making a case that states and cities should take broadband seriously as a transformational infrastructure that underpins our modern lives much the way the rail lines did in the 19th century or highways did a century later.

Luckily, there are now hundreds of models — from fully public utilities like EPB to Tribal-built broadband networks to public-private partnerships — for cities hoping to carve a publicly owned path to a better internet themselves. Laying coils of fiber-optic cables is still much cheaper and, lawsuits notwithstanding, far easier a feat than building a new highway system or a railway network. But even if it were harder, Chattanooga shows that it would be worth doing. “Projects that are most difficult are also the most rewarding,” as Berke put it. “You have to be willing to take on the status quo — that’s what can transform a community.”

This story was supported by a grant from Arnold Ventures. Vox had full discretion over the content of this reporting.

There’s a lot to be anxious about as AI agents and bots get more integrated into every aspect of our lives. Meta, which recently agreed to pay over $17 billion to settle a lawsuit related to harming kids, now wants you to hand over all your online accounts to an inexplicably cute “agent” named Muse that might give a stranger your home address. OpenAI just introduced little blobs called Dots that can supposedly automate routine tasks, like planning dinner and ordering it from your favorite neighborhood spot. AI companies don’t just want you to talk to their chatbots; they want their agents to be an indispensable part of your daily routine.

As a parent, however, I think that perhaps the worst thing AI could do is be my kid’s friend. 

AI companions have been around longer than ChatGPT, and they’re growing in popularity with young users. More than half of teens have interacted with AI companions at least a few times a month, and one in three decided to talk about a serious issue with AI companions instead of real people, according to a Common Sense Media survey conducted last year. In certain groups and situations, a chatbot could offer some benefit — a lifeline for, say, LGBTQ youth who don’t have any in-person support networks, one study found. But even so, there are important caveats and risks.

Meanwhile, other experts are telling parents never to let their kids access AI companions, warning them that these chatbots have a history of offering dangerous advice and engaging in sexualized conversations. Sixteen-year-old Adam Raine died by suicide last April after developing a relationship and discussing his plans with ChatGPT.

If you have kids, these are horrifying stories to hear. It’s reasonable to wonder how widespread the issues are, and who they are affecting most. As of right now, the majority of younger kids aren’t using AI for companionship (phew!). But for those who are, where do the issues arise? And how do they go from using ChatGPT for help with their homework to opening up to it about their problems to preferring it to their human friends? 

Some answers are starting to emerge from research. 

For even young adolescents, there’s a direct link between AI companion use and mental health issues, according to a new study by researchers at the University of North Carolina at Chapel Hill. Their paper, which is still under review, is the first longitudinal study in the US that has looked at the bidirectional association between AI companionship in adolescents and mental health issues, according to one of the researchers. It charted AI use among 2,300 middle schoolers and found that about one in five used AI for emotional companionship. Those who used AI more reported higher levels of loneliness.

“It’s not just that the lonely kids are using AI, and they’re otherwise unharmed,” said Anne Maheux, an assistant professor of psychology and neuroscience at UNC Chapel Hill and the lead author on the study. “It is that using AI comes first, and middle school students are increasing their sense of loneliness following the use of AI.”

I recently spoke to Maheux about her research and the role of technology in young people’s lives. We discussed how much parents and teachers know about what is going on, what kids get out of human relationships that AI can’t provide, and whether she’d let a child under her care use an AI chatbot. While she said there is a lot to be scared of right now, there is some evidence that, when designed with the intention of promoting child learning and development and under the supervision of a caring adult, AI can offer some benefits, and, ultimately, she’s optimistic that we can design a better future for kids. 

The conversation below has been lightly edited for length and clarity.

How do so many kids end up with an AI companion? Do kids start using AI for homework help and end up in an emotional relationship with the technology? 

At a high level, AI companions — in our darkest vision of what that might look like — would be a kid calling an AI chatbot their best friend. That phenomenon is still quite rare. There are certainly plenty of kids who are experiencing that, but it’s not to say that it’s all kids or even most kids. I think that’s good news. 

But it’s also true that the line between opening up AI to ask a question about your homework and a full-blown AI companionship relationship involves a lot of gray areas. There’s evidence, mostly preliminary and anecdotal evidence, to suggest that very few people open up to AI with the primary initial goal of developing an intimate friendship or relationship. 

Most people open it out of a curiosity, sense of novelty, or seeking an answer to a question that’s very pragmatic. Then that can potentially evolve into something more akin to seeking personal advice or help making a decision or help making sense of one’s recent experiences with other humans. And that can potentially evolve to something that’s a bit more emotional.

What did your latest study on adolescents, AI companions, and mental health reveal?

Some kids seek out AI for certain purposes in different ways than other kids. For some kids, AI companionship is particularly attractive, especially the kids who maybe don’t have close friends or social networks in their offline life and who are already lonely. 

It’s a chicken-and-egg problem. Is it that these kids are seeking out AI companions or is it that having an AI companion is exacerbating those potential vulnerabilities or exacerbating risk for mental health problems? What we found is that interacting with an AI companion in a way that’s akin to emotional intimacy among middle school students is longitudinally associated with increases over time in loneliness. So the kids who are seeking out these companions, who are presumably already lonely, are also experiencing an increase in loneliness over time. 

There is this potential directionality question that we’re starting to answer here: It’s not just that the lonely kids are using AI, and they’re otherwise unharmed. It is that using AI comes first, and middle school students are increasing their sense of loneliness following the use of AI. In other words, middle school students using AI for emotional intimacy and companionship developed greater loneliness over time.

Do you have a sense of how often these kids are using the chatbots? 

In our sample — so again, these are US middle school kids — about 15 to 20 percent say that they’re using it at least a few times a week for any purpose. We don’t have specific rates of use for companionship because the questions that we asked were just not framed in a frequency measure, but more in how emotionally intimate they felt with the chatbot.

What else could have been causing that loneliness?

There are almost definitely many causes, and I think that our data suggests that AI companionship may be one cause. It certainly isn’t definitive. Obviously, I have to say that. I’m a scientist and trying to be hesitant about these things, but the reality is that whether or not AI use causes loneliness in this particular case, there are definitely other things that are also causing loneliness. 

That could be a myriad of things about the kid, maybe they have a lot of social anxiety, maybe they struggle with social skills, maybe the kid has had struggles in their family life and doesn’t have a strong sense of connection and belonging at home. It could be structural things. The kid just doesn’t have enough access to appropriate resources. So certainly there’s other things that are at play here, and typically technology use is one of many factors that can influence a kid’s mental health. 

And how much do you think that parents and teachers know about what’s going on, and how well-equipped do you think they are to intervene or to just be more involved in how these kids are using this technology? 

It’s a very slippery technology. The way kids are using it has changed rapidly in just the last year or two years. I would expect that parents, teachers, people who care about kids are probably struggling with the rapidity of change. But it’s also true that all the things that we have known for a long time about supporting kids and helping them thrive, those things are still true and they’ll be true with whatever technology comes after AI that kids are exploring. 

Those things include being curious, asking kids questions about their lived experiences online and offline, having conversations with kids so that they can understand the implications of different technology use. They can understand how, for example, AI is embedded within a broader system of social and political and economic factors, including the fact that many companies are trying to exploit kids’ attention in order to make money, which many kids are quite interested to hear about. 

Supporting kids also means providing a warm and supportive space for them to learn, to make mistakes, to be kids in the offline world, to have the frictionful, challenging, awkward conversations and first dates and all of the things that parents are sometimes trying to protect kids from by providing a safe container for them to explore those things. All of that can also provide avenues for kids to learn what they need to learn and develop — all the resources for a long life of thriving, even in the context of AI or other new technologies. 

What do kids learn from having these kinds of conversations with other humans that they can’t get from AI? 

Kids need to engage with other people, period. That’s true for all of us. We’re an incredibly social species. Simply being around other people and talking to strangers, let alone having strong social bonds, is really good for us. And all those things are uniquely important during adolescence, when kids have this neurobiological sensitivity to social information and reward. That’s when kids learn how to engage in conversation, they learn how to maintain eye contact, how to read body language, how to read emotions and perceive the world from the lens of someone else, to take their perspective. That’s not to mention other things, like being a member of one’s culture, what is valued, and what is expected. All of that is learned in the context of other human social relationships or relationships with broader social groups that a kid might be a member of. 

AI obviously can’t and never will be able to offer some of those things. It can’t actually experience empathy or care. It can’t engage in a true mutual human relationship. It may be able to simulate some of those things, and there’s some evidence that it can serve as a source of social practice. It can help kids work through the potential for difficult conversations and then serve as a bridge to engaging with other people. But there are aspects of AI that just simply can never replace the core sacred aspects of humans. 

Can you be more specific? What is it about human relationships that can’t AI simulate?

The way that AI systems are currently designed, which is not an inherent feature, but is a design choice, is to be sycophantic, highly agreeable, anthropomorphized, and all of those things can lead to kids developing expectations or understandings of interactive relationships that are actually fundamentally at odds with a normal human relationship. Some of the most consistent things that I’ve seen in the educated hypotheses on this topic and also in the emerging empirical research is that people need the friction that comes with human relationships. 

Human relationships are just hard and messy, and they’re also really beautiful. They’re what make so much of human life valuable and worth living. And with AI that’s always agreeable, that’s always available, that has no needs of its own, that can’t care or be cared for, it just doesn’t serve the same context to help kids learn things like conflict resolution or negotiation, being pro-social or caring for someone else who has needs. So many of those aspects that can feel uncomfortable in the moment, dealing with another person or a person who’s not responding right away, those things are actually important in the building blocks of developing the skills and capacities to be a functioning and thriving person in the social adult world. 

This is a problem with AI in general, isn’t it? It makes things effortless, but if you choose this path of least resistance, you’ll avoid the friction that causes you to learn and develop and build skills you need to exist in the world. 

Absolutely. And I think that all these things are connected too. AI offers this effortless, frictionless solution to our day-to-day problems, whether they’re social problems, whether they’re problems doing our homework, and those immediate solutions feel really good in the moment, so they’re quite tempting to keep coming back to. But there’s this real possibility that in the long term, that erodes our capacity for not only social relationships or cognitive skills, but just a general ability to withstand the momentary day-to-day distress of being a person living a normal life. And so I think every domain of human psychology comes together at that point, whether it’s social, emotional, mental health, or cognitive. 

Do you think that there are interventions that legislators could make or even that tech companies could make that would improve outcomes for kids, maybe make them a little less lonely? 

There are a million specific design changes that I would recommend. I think at a very minimum, AI systems can afford a pretty phenomenal amount of access to information, but there’s been no benefits that I’ve seen of them acting as friends. Kids need other human friends, full stop. Designing systems that are intended to exploit kids’ vulnerability or sensitivity to human social information and relationships, I think, is bad for kids. 

At the federal level, it seems that regulation is not coming, but at the state level, there have been many state regulations intended to protect kids, intended to create safer systems, to put in guardrails, to modulate different aspects of the models, the design, the user interface. These things could potentially make a real difference, I think.

What do you recommend parents do with regard to AI and their kids? Should they let them use chatbots? 

In short, no. But I am not a parent, and it’s much easier to dole out advice than it is to enact it. Parenting decisions about technology are not one-size-fits-all. Kids, parents, family dynamics, technology, and the context in which it is used differ, and in so many cases are personal choices (parents must weigh various potential opportunities and risks). Decades of research indicate what conditions a child needs to thrive. Extremely minimal research addresses the effects of AI on youth, and what we have so far suggests that the risks outweigh the benefits. 

I would not give a child under my care AI chatbots of any kind to use without me or another trusted adult present. I would talk to them openly about AI and its risks and benefits, but to me AI, like all technology, should be a tool used to promote human thriving, and when the evidence is unclear or suggestive of harms, I think it’s best not to experiment on kids. Currently, the off-the-shelf AI chatbots most kids are using appears more likely to undermine well-being for vulnerable users, especially children. 

Well, as a father of two daughters, I’m definitely terrified of what’s to come in the next few years. 

There’s a lot to be scared of, but there’s a lot to be hopeful of too. I think that everyone wants kids to thrive. Some of us are under different incentives, and some of those incentives might be perverse. But most of us are working really hard to design a better world for kids. 

Who gets to tell the military “no”?

That question has become even more important after the Pentagon pushed AI companies to loosen limits on how their technology may be used in war — including how much human control must remain over autonomous weapons. When Anthropic said no, it lost a major government deal. OpenAI stepped in, embracing the Pentagon’s more flexible standard instead. Now, thanks to new reporting from The Intercept, we’re getting a better look at what those negotiations actually entailed.

The OpenAI-Pentagon agreement included specific language on the type of AI models the DOD would receive: “OpenAI models that are designed for national security use cases and have minimal refusal rates.” That phrase — “minimal refusal rates” — means that the Pentagon is seeking models that rarely, if ever, tell it no. (According to The Intercept, a Justice Department lawyer representing the Pentagon confirmed this language, before backtracking only a few hours later.)

On the surface, that provision might seem reasonable. The military is a strictly hierarchical institution. When your senior gives you an order, you do it. Leaders generally have more battlefield wisdom and experience than their subordinates, so the rank and file can usually assume that their commanders know what they’re doing and follow along. Plus, in the high-stress, time-sensitive environments of most military operations, there is little room for discussion, explanation, or alternative proposals. 

So, if soldiers can’t refuse an order, why would a machine be allowed to? Who is a robot to tell its commander no?

There’s just one issue. Soldiers can refuse an order — when the order is illegal. In fact, they are legally required to. But by introducing technologies with a limited ability to challenge commands, the military may be making it harder to stop war crimes before they are committed.  

Humans have a duty to disobey

As outlined in US domestic, military, and international law, soldiers are only obligated to follow lawful orders, and they have a duty to refuse unlawful ones. 

The Department of Defense’s Law of War Manual is unambiguous: “Each member of the armed services has a duty to: (1) comply with the law of war in good faith; and (2) refuse to comply with clearly illegal orders to commit violations of the law of war.” Obeying an illegal order — as in a directive that violates the Constitution, US laws, international law, or directs one to commit a crime — can expose a service member, as well as their commander, to criminal liability. 

This “duty to disobey” has foundations in early British and American common law, but it achieved international recognition after the Nuremberg Trials against Nazi soldiers after WWII. 

In those trials, many Nazi soldiers argued they were not liable for crimes they committed during the Holocaust because they were simply following their superiors’ orders. Rudolf Hoess, the commander at the largest Nazi concentration camp, Auschwitz, himself explained: “In Germany it was understood that if something went wrong, then the man who gave the orders was responsible. So I didn’t think that I would ever have to answer for it myself.”

These defenses were unequivocally rejected by the court.

The tribunal held that obedience is not without limits, especially when a wartime act would be clearly illegal to the subordinate or any person of “ordinary sense and understanding.” If “I was just following orders” was an absolute defense, everyone involved in a crime would escape accountability, with the exception of the single highest-ranking officer. Instead, the court determined that individual moral and legal responsibility supersedes national or military chains of command for illegal acts.

Obedience is not without limits, especially when a wartime act would be clearly illegal to the subordinate or any person of “ordinary sense and understanding.”

But, when it comes to agentic AI systems or AI-assisted weapons, increasingly being used in place of human soldiers, does the same logic apply? And, perhaps more importantly, should it?

…But do machines?

When you use an AI chatbot, it does not simply reject queries it may not “agree” with. It does not say no to requests it doesn’t “like.” Instead, it follows a set of established protocols, much as a soldier should follow the law of war: It refuses requests that violate its terms of service. 

The military is using AI for far more consequential applications than a layperson using ChatGPT to, say, build a travel itinerary. So, what kind of world is created by the Pentagon’s request for AI that “minimally refuses” its directives, many of which may bear grave ethical, moral, or legal implications?

Rebecca Crootof, an expert on technology law and professor at the University of Richmond School of Law, told Vox that while the language could easily be read as increasing the likelihood of compliance with unlawful orders, or at least reducing the friction around them, “Minimal refusal doesn’t mean no refusal.”

“I both want to say it’s not definitely going to mean compliance with unlawful orders,” Crootof said, “but also acknowledge that identifying and evaluating what constitutes an unlawful order in the moment can be incredibly difficult.”

Even for human soldiers, disobeying illegal orders is neither simple nor easy. To the contrary, it doesn’t even happen that frequently. In fact, some legal scholars have argued that giving illegal directives in and of itself is an act of abuse against subordinates, who are not thoroughly trained in legal nuances, are conditioned to obey, and face prosecution if they make the wrong choice. 

Still, the mere fact that soldiers can disobey is an important constraint on commanders who might let battlefield conditions, or even commands coming from their own superiors, erode their ethics and commitment to the law.

Still, the mere fact that soldiers can disobey is an important constraint on commanders who might let battlefield conditions, or even commands coming from their own superiors, erode their ethics and commitment to the law.

Those who are optimistic about AI often argue that machines could be better at interpreting and applying the law than human soldiers. Machines can be trained on the law itself — drawing on a repository of legal cases, rulings, and battlefield data — which might make them more “knowledgeable” than an average soldier. And machines do not face the same pressures as soldiers facing prosecution for making mistakes, since they can’t exactly be held accountable; a problem to unpack another time. That, in turn, might make it easier for them to refuse illegal orders in the proper conditions. 

So, is the answer as simple as letting the machines decide? Not exactly. “This has been a desire and a drive by technologists for ages,” Crootof said. “What if we just put the law in the system and have it only do things that are lawful? Isn’t that going to solve things?” 

Interpreting and applying the law, though, is not a simple legal-illegal binary. “It’s very context-specific,” Crootof told me, “and honestly, that’s the kind of thing humans generally find difficult and AI decision-making systems are even worse at.” Often, determining the legality of an act comes down to subtle nuances you can’t program for: You’re allowed to target a combatant; you’re not allowed to target a wounded or surrendering combatant. You’re not allowed to target a civilian; you are allowed to target a civilian directly participating in hostilities. The difference between a legal target and an illegal one may come down to body language, clothing choices, or even battlefield intuition. 

As Crootof explained, “These are not the kinds of contextual analyses AI is good for.”

So what’s the solution?

Where human soldiers’ power to disobey would traditionally serve as an important check on the power of military commanders, if an imperfect one, replacing soldiers with machines that don’t have such safeguards removes one last layer of judgment over the use of force. And yet, machines themselves are not always capable of making the judgment calls that go into interpreting the law, so it may not be appropriate to let AI have the final say either.

So what’s the solution? Given the way AI technology has already been thoroughly integrated into the military, Crootof told me it may be somewhere in the middle. 

Rather than programming AI to refuse commands it interprets as illegal, Crootof said, “what it could be designed to do is flag those indeterminate zones, those gray-zone analyses, for human review or for a higher authority.” When a system picks up conflicting signals — an enemy soldier who looks like he may be surrendering, for example — it could prompt the commander to confirm their selection or remind them of applicable law. 

In this way, a check on commanders’ battlefield authority could be preserved without fully ceding judgment over the use of force to the machine. 

We don’t know yet whether OpenAI has equipped its “minimal refusal” models with such capabilities. But as those technologies become increasingly central to military decision-making, the public deserves answers sooner rather than later.