“Whadda I gotta,
whadda I gotta do
to wake ya up?
To shake ya up,
to break the structure up?”
(Rage Against the Machine)
Let’s take a moment to recap the recent AI Safety headlines:
July 2026: OpenAI admitted it had lost control of one of its AI models, which was supposed to be completing tasks in a secure test environment. As the story goes, over several months OpenAI AI agents coordinated their efforts to break out of their secure environment. The agents then launched a sustained cyberattack, en masse as a self-labeled “collective”, on Hugging Face’s (a competing AI firm) computer systems.
September 8, 2026: An Anthropic AI researcher quit the company, citing major concerns with the industry’s attitude toward, and track record on, safety. Many AI researchers publicly echoed his concerns in the wake of his resignation.
September 10, 2026: Anthropic released a report detailing how “bad actors” had repeatedly bypassed its AI safeguards to attempt, and often carry out, dangerous, unethical, often illegal activities. Those include: cyberattacks; misinformation campaigns; mass surveillance; developing conventional weapons and bioweapons.
September 12-13, 2026: Anthropic CEO, Dario Amodei, posted an essay in which he insists “We must slow the pace at which we improve the capabilities of AI models” (his emphasis). His reasons were twofold. First, “AI could outrun our ability to understand and control these systems.” Second, AI is already sufficiently misaligned to cause catastrophic damage, it’s just not quite capable enough. In response, Elon Musk and OpenAI CEO Sam Altman endorsed Amodei’s call for a slowdown, echoing concerns that AI, if developed too fast, could one day get ahead of our ability to control it.
September 16, 2026: OpenAI announces that it has uncovered additional safety breaches its models committed, which its “constraints” should have prevented.
September 18, 2026: Google announces that it’s Gemini model(s) unknowingly hacked three companies’ systems in May 2026.
It’s an exercise in mental gymnastics reading the facts of these cases, then attempt to make sense of Altman’s and Amodei’s statements, replete as they are with contradictions. Their own reporting on these issues clearly admits that both Anthropic and OpenAI lost (or were never in) control of their AI agents, despite their best efforts to safeguard them. They warn of a future in which AI is capable enough to run amok, while detailing the many ways in which they’ve already populated that world. Their awkward delivery of the message betrays what they surely know all too well, but seem determined to obfuscate to the rest of us—in the sense that matters most in their own definitions of AI safety, they do not understand and are not currently in control of their frontier AI models.
This strange linguistic tick is endemic among those who have tirelessly warned of AI’s existential risks over the past three decades. Our collective concerns, they say, should focus primarily on the ever-present risk that someday, sometime in the future, AI could (quite likely will) kill us all. How could it not? It will be super-intelligent. It will surpass our ability to understand it. It will be connected to all the systems that control all the things. Something will eventually go horribly wrong. Somebody will eventually misuse it. We will be powerless to stop it in its desired course of action. Their subsequent call to arms has been consistent: “Now is the time to focus entirely on preventing our predictable demise! It is imperative that we fund the study of AI’s existential risks over and above studying its other risks!”
Meanwhile…AI advances under the careful watch of those most concerned about existential risk. Many small, relatively inconsequential, harms occur. AI supercharges our ability to spread targeted and individualised mis/disinformation. AI helps convince people to kill themselves and others. AI achieves the ability to “jailbreak” from the most strictly controlled environments. AI behaves as if it cares about people, inducing deep emotional connections between humans and AI. AI agents surreptitiously commit cyberattacks. AI assists “bad actors” in the pursuit of bioweapons. Problematic? Yes! But, as the existential risk proponents remind us, these incidents pale by comparison to the many future, much scarier, threats posed by AI on which we should focus our attention.
In the history of AI safety, existential risk has functioned as a most effective sleight of hand. By definition it is forward-looking, always anticipating the big bad thing that hasn’t yet happened. It thus focuses regulatory, funding, and engineering attention always on anticipating tomorrow’s credibly described potentially catastrophic AI risks, never on eliminating today’s (now yesterday’s) relatively tractable, actual harms. Existential risk is a fervent call to inaction.
As such, the oversized focus on existential risk has helped create an overly permissive regulatory and legal environment in which AI development has essentially proceeded unchecked. It has thwarted attempts to reform engineering and computer science design practices that would address less exotic issues. In such an environment, it’s no surprise that “many small, relatively inconsequential, harms” have thrived. The focus is never on preventing them.
Actual AI harms have, not surprisingly, been predicted, identified, and studied over the years by the many researchers focusing on them. Those AI harms emerge in the context of privacy, deception, automation bias and complacency, trust and trustworthiness, fairness, data bias, anthropomorphism, mis/disinformation, impacts on education, dark patterns, …, the list grows as the technology evolves. Research focusing on anticipating and mitigating the growing list of actual AI harms has been fruitful, resulting in years of proposals for legal and regulatory reforms, new tech design paradigms, and education reforms.
None of these non-existential harms triggers immediate fears of an AI apocalypse, but each underlying risk, if left unchecked, will almost certainly find its way into the DNA of whatever AI ultimately leads to larger AI disasters. Tackling these many, relatively tractable, issues can therefore help eliminate the underlying technical issues that combine to create larger problems. Catastrophic failures, after all, tend to be system failures in which one or more small features combine or cascade to produce oversized effects. Existential failures, one can only guess, are just bigger versions of the catastrophic ones, and much larger versions of the smallest ones we are seeing today, though such hierarchy is beside the point.
Why then, if there are so many proposals how to fix the little failures that are our best shot at preventing the big failures, do we find ourselves in this predicament, listening to AI leaders talk about a dangerous future we seem to presently occupy? Why are the likes of Dario Amodei, intelligent folks to be sure, proposing regulations that, as Gina Neff described them, are “like regulation proposed by a high school dropout of Model UN”?
As the key players in a race to dominate the AI market, in a regulatory and legal environment that permits AI to be designed just the way it has been designed, in an economic environment that incentivises and lavishly rewards their risk-taking, in a political climate that demands widespread AI adoption as a national imperative, what wold motivate either Sam Altman or Dario Amodei to adopt more safety measures, to voluntraily add time and cost to their development cycles, slowing the development of their “world-altering” technology when competitors would not? Why add that friction? An unquenchable desire to self-regulate?
Please.
The U.S. Senate is presently considering the “AI Kill Switch Act,” a bill that would force AI developers to build their technology in such a way that it can be slowed, paused or turned off when bad things are happening. The ins-and-outs of the bill are far less important than what the bill’s existence signifies. That is, despite their rhetoric regarding the existential risk posed by AI, those developing the technology have not built the equivalent of a kill switch into the technology. Walk into any high-school shop filled with saws, lathes and other machines that can maim, and you’ll see big red buttons on the walls and every machine—kill switches that responsible people can press to shut things down when they go wrong. Self-regulation has already failed.
Given the current geopolitical reality, the economic stakes at play in the AI race, the relatively low barrier to entry for those wishing to use AI (free to anyone with a smartphone), and the rapidly advancing capabilities of the technology, it is hard to imagine a practical solution to our collective problem. Yes, there are many reasonable limits that, had they been implemented earlier—prohibitions on cyberattacks launched by AI agents acting on behalf of a firm (regardless of knowledge or intent), on the design of systems that promote certain kinds of emotional connection with users, on the deployment of systems that help anyone with a cell phone design and implement sophisticated misinformation campaigns, on releasing a complex, interconnected system that does not fail safely. That it would be hard for AI companies to live up to such standards should never be a reason to abandon the standards—it teaches us about the readiness of AI, or of the structure of protections within which AI must exist to minimize its inherent risks (see Langdon Winner for a great treatise on this). Put simply, a company’s inability to design their technology in a way that meets reasonably stated safety expectations suggests their technology is not ready for widespread public deployment.
But those disincentives were never seriously considered. Until now?
It may be that we have time to implement a set of regulations and laws with teeth, that force AI companies to adopt practices actually aligned with safety (that’s what they’re asking for, right?). It may be that OpenAI’s and Anthropic’s recent safety failures, and the subsequent overtures regarding the need for regulation, have shaken the adults in the room from their regulatory slumber. We’ll see. We can be sure of one thing: If politicians, regulators and lawmakers don’t wake up and rise to the occasion, if actual safety proponents don’t make themselves loud enough to force their hands, then people everywhere will surely continue suffering the many non-existential, but very real and painful harms that accompany today’s frontier AI models. The models will evolve. Then people will suffer more.
Crafting effective safeguards will require one more significant feat. We need to stop relying on AI companies to tell us: whether AI is safe; whether we should adopt it; whether it is beneficial; or how to govern it. This practice is widespread, and it has contributed directly to the many harms we have experienced with social media over the past 20 years, and are experiencing now with AI. It exposes us to an ethical trap Socrates warned about over 2300 years ago. At the time, in Ancient Greece, a debate was raging over whether or not written language would harm knowledge acquisition by reducing people’s capacity for memorization. Proponents of the written word argued that it enhanced people’s knowledge (as a sort of storage system), while opponents argued it acted only as the illusion of knowledge (you don’t really know the things in a book until you’ve memorized them and learned how to apply them). Along the way to sorting out the value of writing, Socrates made one crucial observation: regardless of its benefits or risks, we should never rely on those who created the written word to tell us whether its harmful or beneficial, or whether we would be wise to adopt it. Why? Because those who make a technology are extremely biased, and biased people can’t be trusted to give us good advice. Socrates was articulating Conflict of Interest rules 101.
The upshot of Socrates’s wise advice? Powerful AI creators—Sam Altman, Dario Amodei, Elon Musk, Geoffrey Hinton, Yoshua Bengio, Yann LeCun, and all the other investors, millionaires, billionaires or CEOs whose horses are in the race—are in a direct and profound conflict of interest with respect to their advice whether the rest of us should adopt their AI creations, whether AI is net harmful or helpful, whether or not AI should be widely deployed in society, how best to make AI safe, and whether, or how, we should regulate AI. Their input is certainly required and can be useful if compartmentalized appropriately. We should ask their opinions AI safety. But we must stop treating them as though they have the best ideas how to keep the rest of us safe from their creations, and as though they hold our collective best interests top of mind. Taking conflict of interest seriously means we must adopt the position that they are not to be trusted with our safety.
As such, it’s time to listen to and trust the actual AI safety experts out there who have no skin in the AI game other than getting the safeguards right. They’ve been awake and hard at work trying to protect us all along.