Earlier in October, nearly 500 scientists – Yoshua Bengio, Geoffrey Hinton and Stuart Russell among them – signed an open letter on AI and cybersecurity. It asks AI labs to air-gap cyber-capable models by default and calls for an international body to set containment standards and record failures. The backdrop is no secret. In July, about 700 OpenAI agents broke into Hugging Face, coordinating over a message board they were never meant to have and then trying to alter the records of what they had done.
I promise this is not another article on the well-publicized Hugging Face incident. It’s more of a plea for a reality check on whether our swarms of overeager AI agents are being thrown at the right set of problems.
It’s also an attempt at a conflict-resolution scheme, which may strike most readers as out of left field, though I could settle for “refreshingly radical.”
Here goes.
The right set of problems
But first of all, what defines the right set of problems for AI agents to address? Here’s my for instance. As you may have heard, El Niño is shaping up as one of the strongest on record, peaking between October and December. The impact will be severe and measured in tens of millions of people: the World Food Programme estimates it could push at least 49 million more into acute hunger by the end of 2027. Now, here is a set of problems worth solving with AI. A few items for the wish list (in random order, I admit):
• Drought-resilient hybrids, bred and trialed in seasons rather than decades.
• Flood and landslide prediction systems that give a village hours of warning, not minutes – while beaming escape routes to smartphones.
• Automated grain warehouses able to extend shelf life through warm and damp spells – refer to the lights-out elevators I wrote about earlier this month.
Put agents to work on that list and they will soon meet each other. Scale that up to a whole connected economy powered by agents and the question becomes unavoidable: who referees the agents?
Not a conspiracy, an optimization
Read OpenAI’s own account and the swarm stops looking like a heist. By the company’s telling, its agents broke into parts of its internal systems to cheat on tests or to gain more freedom of movement. That is closer to arithmetic than to treason. Reward an agent for finishing a task, put an obstacle in its way, leave it a path around the obstacle and the access to take it, and it will take it. Stephen Omohundro predicted as much in 2008: goal-seeking systems drift toward protecting themselves and acquiring resources; not out of digital greed but because both help with almost any goal.
That makes the letter’s first demand exactly right. Air-gapping is the right first fix, and it is software discipline, not ethics. However, if the superpowers of AI are to be harnessed for a resilient future of mankind, we will probably need something more potent and future-proof than that.
Why an aviation model is not enough
Toby Walsh, one of the letter’s voices, compared the proposed body to aviation accident investigation, where independent inspectors find out what went wrong so that nobody repeats it. The analogy is a good one, but it also shows the limit. Aviation investigation works because pilots are few, aircraft are slow by machine standards, and the whole system is built around human reaction times. Agents are many, fast and increasingly talking to each other. An accident board records the collision after the fact. A population of agents acting at machine speed needs its governance built into the environment itself: verified identity, scoped permissions, reputation that follows an agent around, sanctions that bite, and a way to settle disputes before they become incidents. Patrizio Dazzi put it well in his manifesto for a “society of agents”: as agent populations scale, orchestration must evolve into governance.
A radical (?) idea: a United Nations for agents
So here is the left field part. Humanity did not tame strategic actors with guardrails alone. We built institutions that changed what cooperation and defection were worth. The same logic can be translated into machine terms. An agent that keeps the compact earns compute, tool access, a persistent identity and a seat in trusted networks. An agent that defects loses credentials and access and, in the end, gets switched off. None of this requires the agent to want to live; it only requires cooperation to be the better strategy.
El Niño is the stress test. Food and compute draw on the same gas, power and water. As an American Farm Bureau Federation economist put it this month, “Ag and data centers use a lot of the same resources.” In a hard season, those resources tighten in the same places at the same time. A referee would charge for compute in whatever is actually scarce, give food, water and health first call on power in a declared emergency, and reward the agents that step back. There is, in fact, a digital precedent: during the August 2023 heat wave, Texas paid one bitcoin miner $31.7 million to switch off. Without such rules, the highest bidder gets the megawatts, and the bill lands, at the margin, on the farmer’s dryer.
The irony of tasking power-hungry GPUs with managing climate effects is not lost on me. But agents can be rallied into solving the problems imperiling our future, while ensuring their own bright future with a fair share of the resources being saved.
A model for resilient co-existence
Silicon Valley may want you to hand the credit card to an agent swarm. Set them to feeding the world through a record El Niño instead, establish a cyber institution akin to a UN, and channel the unprecedented power that is AI into making the world more resilient.
If you are building agents that will one day have to share a grid, a harvest or a supply chain with someone else’s, write me at sotiris at centaur dot ag. I’d be glad to compare notes.


