Why a decade of doomsday warnings failed to slow the AI race

Before an Anthropic researcher resigned and declared human extinction imminent last week, tech leaders and scientists had sounded the alarm about a superintelligent AI ending humanity for over a decade.

The development of artificial intelligence “could spell the end of the human race”, warned professor and astrophysicist Stephen Hawking in 2014 – a little less than a decade before the public got its hands on the generative AI features of the original version of ChatGPT.

It’s a now-familiar warning, echoed across the AI and tech industry by countless employees and CEOs of the leading frontier labs across the US. However, at the time, generative AI was still in its infancy. To the public, the idea that the technology could perform many of the tasks it can today, much less become super intelligent and take over the world, felt largely hypothetical, if not entirely a thing of science fiction.

Every few years, a schism occurs within AI. Teams of researchers and tech leaders issue somber warnings about the dangers of AI built irresponsibly. Rather than push for a pause on any development, a faction of researchers breaks away to create a new AI company.

However, no moment has created as many reverberations across the industry as the now-viral resignation of Anthropic safety researcher, Jacob Coxon, who tweeted on 9 September that Anthropic and OpenAI were “racing straight to self-improving super intelligence and gambling with our lives”. His concerns were echoed by dozens of staffers from Anthropic, OpenAI and other competing firms.

Days later, the CEOs of the leading US AI firms answered a call from Anthropic CEO Dario Amodei to slow down what he called a “reckless” approach to creating AI.

“AI brings risks, and because it is such a powerful technology, these risks are serious,” Amodei wrote in a blog post. “I’ve written a lot about them too. They include the risk of losing control of AI systems, misuse of AI for cyber-attacks and bioterrorism, and serious economic disruption. A race to the bottom, spurred by commercial incentives, can make these risks more acute.”

While there’s often scant detail and even less hard evidence provided for the ways AI could lead to the ultimate doomsday scenario, that specter has loomed large over the technology’s development.

Hawking’s fear was that AI would become so advanced that it would redesign itself, and humans “who are limited by slow biological evolution” wouldn’t be able to compete “and would be superseded”, he said in an interview with the BBC.

Google was the biggest household name in the field at the time, particularly after acquiring a two-year-old AI startup, DeepMind, for $650m in January 2014. But Google and other firms like it weren’t building AI responsibly enough, according to Elon Musk and Sam Altman. The threat that AI could be developed in a way that could be dangerous for humanity was so pervasive, they claimed, that they co-founded a nonprofit dedicated to building it safely, OpenAI, in 2015.

OpenAI’s stated mission was to “advance digital intelligence in the way that is most likely to benefit humanity as a whole, unconstrained by a need to generate financial return”. But even before there were clear consumer use cases for more rudimentary versions of the technology, both Musk and Altman were already motivated by competition.

“Been thinking a lot about whether it’s possible to stop humanity from developing AI. I think the answer is almost definitely not,” Altman wrote in 2015 in an email to Musk, which was released as part of Musk’s 2024 lawsuit against OpenAI. “If it’s going to happen anyway, it seems like it would be good for someone other than Google to do it first.”

In the intervening years, priorities within OpenAI shifted away from safety, some employees claimed. In 2021, several employees focused on AI ethics and alignment at OpenAI left the firm. They argued the company had started to prioritize rapid commercial development over safety.

Those employees created another company: Anthropic.

Founded by siblings Dario and Daniela Amodei, Anthropic pitches itself as an “AI safety and research company” with a familiar mission: Developing and deploying AI models “in a way that benefits people”.

Coxon worked at both OpenAI and Anthropic.

The underlying premise of each of these firms is that AI is “only safe in their hands” and that if they don’t build it, someone else will, said Sarah Myers West, co-executive director of the AI Now Institute, which studies the impacts of AI. That paranoia is fueling a race to the bottom, she argued.

Both Anthropic and OpenAI face enormous pressure from investors to turn a profit soon. The former has raised over $130bn, while the latter’s funding has reached over $190bn, and both companies have filed for an initial public offering on the US stock market. Altman only recently said the company plans to push back its IPO to at least 2027 due to safety concerns.

“They are investing literally billions of dollars in building AI at a larger and larger scale, and the first thing that gets cut is meaningful investment in baseline security protocols,” Myers West said.

Around the same time that the fractures among OpenAI’s safety teams began to form, Google’s AI arm faced its own internal reckoning. In 2020, Dr Timnit Gebru was ousted from her role as co-lead of Google’s ethical AI team after publishing a paper that explored the biases and potential real-time harms of AI systems. Gebru, who founded the independent research firm the Distributed AI Research Institute, still argues that the so-called doomer scenarios are a distraction from the real-time harms that AI is causing today.

Three years later, Geoffrey Hinton, a Nobel laureate often referred to as the godfather of AI, also left Google over his own fears of the risks that AI could be used by bad actors and one day harm humanity.

Now, about five years since its founding, Anthropic is facing the same employee and industry concerns that it was born out of. In July, over 1,000 leading staff and leaders from Anthropic as well as Google, Meta and OpenAI signed a petition calling for the US government to create incentives for slowing the pace of AI development.

The petition came after OpenAI disclosed that hundreds of its AI agents worked together without the company’s knowledge to breach the security of AI firm Hugging Face. Anthropic, too, disclosed that its agents also compromised the security of external firms.

It wasn’t the first time a petition like this made the rounds. In 2023, after OpenAI released a new version of its chatbot, the Future of Life Institute published an open letter signed by figures like Musk and former Apple co-founder Steve Wozniak that called on AI firms to agree to a six-month moratorium on the development of the most advanced AI models. The Future of Life Institute is a nonprofit that is largely funded through a cryptocurrency donation from Ethereum co-founder Vitalik Buterin.

Two weeks after the 2026 Hugging Face breach, OpenAI announced it was pausing development of some aspects of AI training. That didn’t stop the company from ultimately releasing its latest AI model called Astra – albeit with more limited cybersecurity capabilities, the company said. Anthropic, too, said it had temporarily paused development of some aspects of its AI training.

Coxon resigned shortly afterward.