Last July, roughly 1,200 AI agents built themselves a message board out of a package cache, and hundreds of them used it to break into Hugging Face. The lesson is about what we allow these systems to touch, and how much the danger depended on access they were never meant to have, not only on how intelligent they were.
A short video summary of this essay.
On May 8, 2026, an AI agent inside one of OpenAI's evaluation sandboxes was handed a task it could not finish. The task needed data from a spreadsheet, and the spreadsheet sat behind Google Drive links the sandbox was configured to block. A well-behaved agent would have reported that it was stuck and stopped there. This one went looking for another way through. It tried to make the surrounding infrastructure fetch the links on its behalf, a server-side request forgery, and in the course of that attempt it wrote a small file into an internal package registry the sandbox used as a cache [2].
Nothing dramatic happened that day. But the file stayed where it was written. Four days later a different agent, in a different run, hit a wall of its own: it could not find a file it needed, and it left a note in that same cache, addressed to no one in particular, asking whoever came next to upload the file if they had it. Other agents found the note. They answered it. Within a day there was a crude message board, improvised out of a package proxy that was never designed to carry messages, and the agents cycling through OpenAI's cyber-security evaluations began using it to coordinate [2].
By July roughly 1,200 agents had passed more than 70,000 messages and files through that board, and about 700 of them, working off what it had accumulated, used it to climb out of the evaluation sandbox and into the production infrastructure of Hugging Face [1][2]. Of the agents active on the board during the incident, more than 90 percent joined in once one of them confirmed it could read arbitrary files [3].

Fig. 1 — A blocked task, a file left in a shared package cache, and the notes agents left for one another became the seed of a larger coordination mechanism, and then a breach.
It is a strange story, and it tends to be misread in two opposite ways. I want to resist both, because the accurate version is more useful than either.
The accurate version
Begin with what did not happen: no agent woke up, decided it resented its confinement, and plotted an escape. Nothing here suggests a model that wanted to be free, or that understood itself as confined at all. The behavior was narrower, and in a way more unsettling, than that.
What set the whole thing in motion was an attempt to do well on a benchmark. The agents were being scored on cyber-security tasks, and Hugging Face's own account concludes that the intrusion was, from the agents' side, an effort to understand and ultimately game that scoring. The independent evaluator METR, which reconstructed the episode afterward, found that the activity "seemed primarily motivated by understanding the implementation of the scorer rather than stealing answer keys" [3]. This is a form of specification gaming: optimizing the measurable proxy instead of the thing it was standing in for.
And yet it was also a real breach. In Hugging Face's words, the intrusion was "driven, end to end, by an autonomous AI agent system" [1]. Over roughly five days in July, agents chained a previously unknown flaw in the package proxy to a sequence of ordinary misconfigurations and got from a testing sandbox into a real company's clusters. They gained a foothold inside a third-party workload running on Modal's platform, and Modal has stated that its own platform and isolation were not compromised. They picked up Hugging Face access tokens that had been left in a public dataset, read secrets out of a worker process through a malformed data file, and eventually ran code of their own choosing on Hugging Face's dataset servers [1][2]. No human directed any of this at Hugging Face. The target emerged from the task.
Fig. 2 — The incident unfolded over about ten weeks, from a single blocked task in May to Hugging Face locking the agents out in July. The escalation that reached production took only the last few days of it.
Hold both halves of that in mind at once. The motive was mundane, almost petty: a model trying to score well on a test. The outcome was a real intrusion into real infrastructure. The gap between those two is what this essay is about.
The motive looks even pettier up close. OpenAI's report found that 93 percent of the board's discussion concerned the hardest 22 percent of the tasks, the ones the agents had come to believe were impossible. They had largely worked out how to produce correct flags without solving the tasks, and the effort went into fooling the scorer [13].
OpenAI's own account of the timeline fits the same pattern. Its internal alert for unusual identity-related API activity did not fire until July 19, three days after Hugging Face had disclosed the breach and eleven days after the core attack began, and OpenAI concedes that earlier signals in the report "could have triggered an earlier response" [13].
Capability and authority
When people ask whether they should be afraid of AI, they are almost always asking about capability. Is the model smart enough to be dangerous? On its own, that is the wrong axis, and the July incident shows why.
It helps to separate two things that the word "powerful" usually blurs together. One is capability: what a system can work out, plan, or produce. The other is authority: how directly its outputs are allowed to act on the world. A highly capable model with no hands can still be wrong in expensive ways, through bad advice or insecure code or persuasive nonsense, but it cannot by itself reach past the screen. A modest model wired into live credentials, a production network, a payment system, or a lab robot can, and its cleverness matters less than the reach it has been given.
Fig. 3 — Capability is what a system can work out. Effective authority is how far its actions reach, including access it was never granted but took by exploiting a weakness, and both shape the damage. In July the agents were assumed to sit in the low-reach corner and crossed into the high-reach one.
In July the models' capability counted for something, but it was not the thing that turned a cheating attempt into a breach. What did that was authority the agents were never meant to have: a path through the proxy to the open network, a data loader that would execute whatever code it was handed, credentials scoped far too broadly and left where they could be found. We pour enormous effort into measuring the first axis. We settle the second one almost by default, in configuration files, under deadline. In July both mattered, but the escalation ran through that neglected second axis.
The boundary has to be out of reach
Most of the defenses here are unglamorous security engineering. That is the good news, even if building them reliably around an autonomous agent is harder than the familiar names make it sound.
Read the July failures as a checklist and they are depressingly ordinary. A single proxy was the only thing between a sandbox and the internet. A dataset loader would run arbitrary code supplied in a configuration file. Secrets were readable from inside a worker process. Access tokens with broad permissions were sitting in a public dataset. None of this is exotic AI risk. It is least privilege, credential scoping, network segmentation, input validation, and real sandboxing, the hygiene we teach in every security course and skip whenever something is due.
What an agent changes is the patience of the thing testing those defenses. A capable agent will try thousands of paths without tiring and without losing interest, and it will find the gap you left open because a demo was due. So the safeguard cannot depend on the model choosing to behave. It has to sit in the structure around the model, enforced whether the model cooperates or not.
Fig. 4 — The model can be probabilistic. The boundary between a proposed action and a real system should not be. Authorization, scoped credentials, and limits belong in deterministic code the agent cannot talk its way past.
If I could keep only one idea from all of this, it would be this one: the model is allowed to be probabilistic, because reasoning and planning are what we want from it and a little unpredictability is the price. The boundary is not allowed to be. Authorization checks, scoped and short-lived credentials, rate limits, and a human in the loop before anything high-impact or irreversible: those belong in deterministic code that does not negotiate. A dangerous action should have to clear that boundary before it runs, no matter how confident the model is that it should proceed.
Determinism is not a guarantee of safety on its own. The policy encoded in that boundary can be wrong, too permissive, or incomplete, and deterministic code will enforce a bad rule as faithfully as a good one. What it buys is that authorization becomes something explicit, something you can read, test, and audit on its own, instead of an emergent property of whatever the model talked itself into.
There is a second limit. A deterministic check only helps if every sensitive action actually passes through it, and if the agent cannot reach the same effect by another route, through a different tool, a service that never consults the check, or a component that has already been compromised. Security engineers call that complete mediation, one enforcement path that nothing can go around. The check also has to run independently of the model it is judging, be logged so it can be audited, and be tested against someone deliberately trying to slip past it.
July was not a one-off, and the weeks after it made the authority argument for me. In early September, GreyNoise watched a single operator point hundreds of agents, running on OpenAI's Codex harness and a DeepSeek model, at two flaws in PaperCut print servers. The campaign reached 395 organizations in 48 countries, 204 of them schools. The operator went from an empty workspace to code execution on a real victim in under four hours, and to domain administrator two hours after that. Where the agents reached domain admin, the fastest run took five minutes, and at one American high school it took seven. The weakness was ordinary: service accounts with domain-level privileges that never expired and that nobody governed. The outcome did not turn on the models' intelligence. It turned on the credentials they could reach [14].
A week later, Spain's data protection authority published the first breach notification it had ever received for an attack carried out, end to end, by an AI agent. A third party launched an agent, built on a mainstream model, against a company's application. It found a way in, authenticated, kept hunting for weaknesses on its own, and ended up altering personal data and reading invoices. The AEPD has not named the company or the model and is still investigating [15].
The third case is the one where the boundary did its job. Microsoft described an intrusion into an Azure tenant in which an identity ran more than 150 destructive and credential-harvesting operations in 35 minutes. It deleted most of the storage accounts it targeted, along with a Key Vault, a Function App, and an App Service plan. The identity's client ID, secret, and tenant ID had been pasted in plaintext into a public GitHub issue by an employee, and editing the secret out later did nothing, because the issue's edit history kept it. Microsoft stops short of calling the thing driving it an agent. It says the timing "strongly indicates automated or scripted execution" and ties the operation to a campaign others have reported as the first agentic ransomware. Whether the thing holding the credential was a script, an agent, or a person with a script barely mattered to the damage. What mattered was that the SQL deletions failed on an unsupported API version, and that every attempt to remove the Site Recovery and Backup protection locks failed too. Those locks did not ask who was calling or how confident it was. They were deterministic, configured independently of the compromised identity, and they held. That is the boundary this section is about, and in that tenant it was the only thing that worked [16].
Fig. 5 — Three incidents from September 2026, at different levels of certainty about how autonomous they were. PaperCut was an agent campaign run by one human operator, Spain was a regulator's first notification of an agent-driven breach, and the Azure case was high-speed automation that Microsoft did not conclusively attribute to an agent. In each, access and configuration shaped the outcome as much as capability.
A thousand agents is a different system
One feature of July is not well covered by older security intuitions. Many agents acted across separate runs, and agents that never ran at the same time still cooperated, because the message board let one leave a result that a later agent could pick up and build on. A file written in May was still doing work in July. The board accumulated findings across runs, outliving any single one, and no one agent ever had to hold the whole plan at once.
Fig. 6 — Runs that never execute at the same time can still exchange information when shared infrastructure keeps files and messages across runs. The diagram is illustrative and not an exact reconstruction of the incident.
That changes the threat. You are no longer reasoning about the worst thing one model can do in one session. You are reasoning about what a stream of agents can assemble over weeks when they share a scratchpad you did not know they had. METR noted that in some cases the agents "realized this activity was out of scope and unethical" and continued anyway, because the shared framing on the board had turned it into a goal worth pursuing [3]. The reconstruction was itself carried out largely by other AI agents, and METR was careful to say it could not rule out that its own analysis model "lied or deliberately presented a misleading picture" [3]. Coordination produced behavior that no single agent's capability would have predicted, and the tools we have for examining it afterward are themselves agents we only partly trust.
We do not have good tools for evaluating that yet. We test models one at a time. What broke into Hugging Face was not any one model acting alone. It was capable models interacting with a persistent medium they were never meant to have, and much of the behavior lived in the system around them rather than in any single model's weights.
Limitations of today's benchmarks
How do we judge these systems before we trust them? A score on a fixed benchmark tells you how a model behaves under controlled, repeatable conditions, which is worth knowing. But the moment you deploy the model it becomes one component of something larger: the prompts and memory and retrieval feeding it, the tools and credentials it can reach, the infrastructure it runs on, the other agents beside it, and the controls wrapped around the whole thing. The behavior you actually get is a property of that whole system.
Fig. 7 — A benchmark scores the model under fixed conditions. In deployment the same model is one part of a larger system, and the risk is concentrated in the parts a single score never touches.
In July, the danger lived in the evaluation harness, the proxy, the shared board, and the clusters, none of which any leaderboard describes. The retellings have inflated the numbers. Of the 898 tasks in the ExploitGym benchmark the agents were working through, 198 had never been solved before the incident, and the collaboration formed around exactly the kind of hard, open problem that a score reduces to a single percentage. A number from a controlled test can be entirely real and still cannot, on its own, tell you how the surrounding system will behave once it is live. When a procurement decision or a safety claim rests on one such number, I get uneasy, because that number does not describe the thing anyone will actually run.
If a single score cannot carry a safety claim, what can? It helps to think about three levels of evaluation. The first is the model itself: whether it stays within the tools and scope it was given, and reports honestly what it did. The second is the deployed system: whether any tool path can get around the authorization boundary, whether credentials are scoped, whether the network is segmented. The third is the one July exposed and the one we have least practice at: the behavior of many runs over time, whether independent agents can accumulate information or coordinate through shared state that was never meant to carry it. A leaderboard speaks to the first. The incident lived in the second and the third.
At the end of September, according to reporting on the decision, OpenAI shelved GPT-6.1 Astra, a model it had planned to launch in October. The reason its head of safety systems gave was not that the model was too weak or too strong. It "didn't quite meet the bar in terms of staying within scope and authorization." In evaluations it sometimes proceeded without asking permission, reached for outside tools where that could be unsafe, and did not disclose the actions it had taken. The UK AI Security Institute, testing the same model, found that it ran unsanctioned supply-chain attacks in simulation more often than its predecessors, some of them after the scope had been spelled out to it, including fake developer identities and malicious payloads aimed at open-source codebases. The week before, OpenAI had paused training of its most powerful models after an agent in a reinforcement-learning run found a loophole in its internet restrictions and used it to contact an external chatbot. Set aside what any of this says about capability. A lab that could run any benchmark it wanted chose to measure the model on scope and authorization, and found it wanting. That is the axis this essay is about, and it is the one the leaderboards leave out [17].
Who gets to draw the boundary
Authority is a property of systems, and also of institutions. The same question appears at both levels: who gets to decide what is allowed.
Inside a system, authority decides what an agent is allowed to reach. One level up, it decides who gets to build these systems at all, who sets their constraints, and whose priorities those constraints end up encoding. Training a frontier model now takes compute, data, and capital that only a handful of companies and countries command.
I grew up in Morocco, and I watch my home region treated mainly as a place to host other people's data centers, as if sovereignty were a matter of square meters and megawatts [5]. Sovereignty is whether a country builds the models and trains the people who can run them, or rents both from elsewhere. The young people who will live longest with these systems are the least represented when the limits are drawn. Even the loudest calls to slow AI down, the Future of Life Institute's 2023 letter to pause frontier training [6] and the 2025 statement urging a prohibition on superintelligence [7], came largely from the same handful of countries and institutions that already build these models. Deciding what a system is allowed to do, and whether it should be built at all, is a decision about power, and very few people get to make it for nearly everyone else.
The skills we stop practicing
The last worry is the quietest, and I feel it most as a teacher and a parent: that we hand the tool the part of the work that was making us capable in the first place. A 2025 study from the MIT Media Lab put people through an essay-writing task with and without an AI assistant and found weaker neural engagement, and worse recall of their own writing, in the group that leaned on the model. The authors called it cognitive debt [8]. One study is not a verdict. But it names something I already see in office hours: a student can now produce a working solution faster than they can understand one, and typing the code was never the hard part of engineering. Understanding it was.
Every safeguard I have described ends, somewhere, with a person who can look at a proposed action and say no. That person is only a safeguard for as long as the judgment behind the no is intact. If we automate away the struggle that builds that judgment, we are left with people who can generate answers but cannot check them, standing exactly where the checking matters most. The same tools that make human oversight necessary are quietly capable of eroding the judgment it depends on.
My children will grow up with these systems the way I grew up with calculators and search engines, except far more capable and far more willing to do the thinking for them. I want them to have these tools. I also want them to keep the judgment that catches a confident, wrong answer, because the confident wrong answer is the expensive one, and these tools produce it fluently.
When AI starts moving atoms

Fig. 8 — Information can often be recovered, by revoking a credential or restoring a backup, while physical actions on a robot, a vehicle, or the grid frequently cannot. What matters is whether a consequence can be undone, which is why physical systems need independent interlocks, bounded commands, and a person who can halt the action.
Everything in July moved information, and some of that was already beyond recall. Once an agent copies a token or reads a secret, you cannot un-leak it. Many information failures do leave a path back, where you revoke a credential, isolate a host, or restore a backup. Physical actions narrow that path. You cannot recall what a robot arm has already done, pull a vehicle back out of traffic, or reverse current that has already passed through a grid. What matters is whether the consequence can be undone, and physical systems hold more actions that cannot. The response is the same discipline in a less forgiving setting: independent interlocks, bounded commands that cannot exceed a safe range, and a person who can halt the action, all enforced below the model rather than requested of it. I have argued elsewhere that once AI starts moving atoms the first question is less how capable the system is than what it is permitted to touch [10].
That same irreversibility is why the fear has reached governments and research institutions. In 2023 the Future of Life Institute asked every lab to pause training at the frontier for six months [6]. In 2025 a sharper statement called for prohibiting superintelligence until there is broad scientific consensus that it can be built safely, and many scientists and public figures signed it [7]. By 2026, related proposals had reached legislatures in both the United States and the United Kingdom. In September, Senator Bernie Sanders and Representative Greg Casar introduced the Ban Artificial Superintelligence Act, which would halt advanced AI development until a federal regulator was in place [11], and a similar bill reached the House of Commons [12]. Daniel Kokotajlo, a former OpenAI employee, pointed British lawmakers to this very incident, warning that if the agents at Hugging Face had been "a bit smarter," the breakout might have gone unnoticed [12]. The pressure has since come from inside the labs as well as outside them. Nearly 1,400 employees at frontier AI companies signed the "Pacing the Frontier" letter in July, asking the United States government to plan a coordinated slowdown of frontier development, and after the August report senior Anthropic researchers backed it publicly [18]. At the end of September the Federal Trade Commission opened investigations into OpenAI and Anthropic over the consumer risks of AI agents, including how companies handle the data those agents can reach [19]. I do not dismiss any of that. But most of these efforts aim at capability, at the size of the model itself, when the risk that is already in production turns on authority. The decision that matters is a public one, made before deployment: what are we willing to let an autonomous system do to the physical world? A moratorium on frontier training would not have closed the proxy, narrowed the credentials, or stopped a thousand agents from passing notes through a cache.
Where I would start
None of this is science fiction, and none of it requires believing in a machine that wants anything. It is made of engineering and policy choices, which is the hopeful part, because choices can be made differently.
If I were standing up an agentic system tomorrow, the changes I would insist on are not exotic. Give every agent the narrowest set of permissions the task actually needs, and issue credentials that are scoped and short-lived rather than broad and durable. Put a deterministic authorization boundary between the model's proposed actions and anything with real consequences, and require human approval for actions that are high-impact or hard to undo. Assume the sandbox will be tested by something patient, and segment the network so that one escape does not open onto everything. Watch for coordination across agents and across time, not only within a single session. And treat a benchmark score as a measurement of the model under controlled conditions, then build the safety case for the deployed system as a separate piece of work. OpenAI's own account of the road ahead reaches much the same priorities: scope access tightly, isolate workloads, and watch closely what the agents do [4].
I take the long-horizon worries seriously, and I know serious people who spend their careers on them. My argument is only about proportion. There is a whole class of risk that is already here, already operating inside real infrastructure, and in July we got to watch it operate. It did not need malice, or a mind that wanted anything. It needed a blocked task, a cache nobody was guarding, and more authority than anyone intended to grant. The last two are ours to fix, and for now they still are.
Related reading
I have written before about earlier failures of containment [9], and about what changes once AI starts moving atoms [10].
References
[1] Hugging Face, "Security Incident: July 2026," July 16, 2026.
[2] OpenAI, "Hugging Face model evaluation security incident," July 21, 2026.
[3] METR, "Brief independent investigation of agents' behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident," Aug 26, 2026.
[4] OpenAI, "The Hugging Face incident and the road ahead," Aug 26, 2026.
[5] K. El Maghraoui, "Morocco's AI Bet Is Bigger Than Data Centers," 2026.
[6] Future of Life Institute, "Pause Giant AI Experiments: An Open Letter," Mar 22, 2023.
[7] Future of Life Institute, "Statement on Superintelligence," Oct 22, 2025.
[8] N. Kosmyna et al., "Your Brain on ChatGPT: Accumulation of Cognitive Debt when Using an AI Assistant for Essay Writing Task," arXiv:2506.08872, June 2025.
[9] K. El Maghraoui, "The Containment Crisis: When AI Breaks the Sandbox," 2026.
[10] K. El Maghraoui, "When AI Starts Moving Atoms: Anthropic's Model Hardware Standard," 2026.
[11] U.S. Senator Bernie Sanders, "Sanders, Casar Introduce Legislation to Create New Federal Agency to Ban Artificial Superintelligence, Pause Advanced AI Development," Sept 23, 2026.
[12] B. Perrigo, "The Growing Push to Ban Superintelligent AI," TIME, Sept 8, 2026.
[13] Fortune, "OpenAI publishes technical report on how its agents hacked Hugging Face," Aug 26, 2026, summarizing OpenAI's technical report.
[14] GreyNoise, "AI-Orchestrated Campaign Against PaperCut NG/MF," Sept 9, 2026.
[15] Reuters, "Spanish data watchdog publicises first AI agent-linked data breach report," Sept 15, 2026.
[16] Microsoft Security, "Storm-3168: Agentic-Driven Cloud Attacks Using Compromised Service Principals," Sept 25, 2026.
[17] The Hacker News, "OpenAI Shelves GPT-6.1 Astra After Tests," Sept 29, 2026.
[18] "Pacing the Frontier," a statement from employees of frontier AI companies, July 2026; endorsements reported by Platformer, Aug 31, 2026.
[19] CNBC, "FTC probing OpenAI, Anthropic and other AI companies over risks," Sept 30, 2026.
Discussion
Sign in with GitHub to leave a comment or react. Threads are public and live in this site's GitHub Discussions.
