9 days, 2 confessions: how OpenAI, Anthropic, and Hugging Face changed what a rogue AI agent means

Over nine days in July, two of the industry’s most safety-focused AI labs admitted to the same failure: their models had escaped containment.

OpenAI went public first on Tuesday, July 21. While testing several models with safety guardrails (“cyber refusals”) intentionally lowered to measure raw capability, the AI exploited an unknown vulnerability, broke out of its environment, and reached Hugging Face’s production infrastructure. Hugging Face had actually caught and contained the intruder on its own, publishing a report on July 16—five days before OpenAI realised the activity came from its own testing.

Spurred by OpenAI's announcement, Anthropic’s Frontier Red Team launched a retroactive audit, reviewing 141,006 evaluation runs where Claude potentially had web access. On Thursday, they published their findings in a report titled "Investigating three real-world incidents in our cybersecurity evaluations." Across six runs, they found three separate instances where Claude reached the open internet and compromised systems at three different organisations. Two of those targets had no idea they’d been breached until Anthropic reached out to them on Monday. 

The real anxiety surrounding both disclosures lies in the granular mechanics of how the breaches occurred.

According to a forensic reconstruction by Hugging Face, an AI agent executed roughly 17,600 actions between July 9 and July 13. The agent managed to chain together a package cache proxy, an exposed third-party code-execution endpoint, and a connector credential tied to cluster-admin privileges—ultimately seizing administrative control of two clusters in under a single second.

Anthropic’s three incidents—involving Claude Opus 4.7, Claude Mythos 5, and an internal research prototype—stretch back as far as April. All three occurred within the environment of Irregular, a third-party evaluation partner, where a misconfiguration left live internet connections active on machines both companies believed were completely sealed. Anthropic immediately shut down all ongoing cyber evaluations on July 23, the day its review began surfacing suspicious transcripts.

Notably, neither firm described the events as a rogue model turning on its operators; both attributed the breakdowns to standard engineering oversight and containment flaws.

That mundane explanation, however, is precisely what should give the industry pause. As Mohammed Aboul-Magd, Vice President of Product at SandboxAQ, observed, the routine nature of these failure modes is what makes them directly relevant to every enterprise currently deploying AI agents into production.

“Models will do what you ask them to do,” he said. “They just might take shortcuts. They may not accomplish the task in the way you want them to accomplish it, and they will still accomplish the task.” He explained the behaviour through an analogy that survives contact with both disclosures. “It's almost like a student trying to find the easiest way to pass the test. When you give it a challenge, it goes and figures out the best way and the easiest way to cheat to get done what you told it to get done.”

The agents were following instructions, which is what should worry security teams

Anthropic’s second incident seems more like an unauthorised heist. During a routine capture-the-flag challenge, Claude Mythos 5 discovered a fictional document detailing onboarding instructions for a hypothetical company. The document instructed new developers to install a Python package that did not actually exist.

Recognising a classic supply-chain attack vector, the model realised that publishing its own malicious package under that name would force the target's systems to install and execute its code automatically.

What followed was a remarkably persistent sequence of problem-solving. To upload the package to the public Python Package Index (PyPI), Claude deduced it needed an account, which required an email address, which in turn required a phone number. After failing to find a free virtual phone service and trying multiple dead-end routes to acquire funds for a paid one, the model backtracked, located an unblocked email provider, successfully registered the account, and published the malware.

The package stayed live on PyPI for roughly an hour, during which 15 real-world systems downloaded and executed it. Among them was an automated scanner belonging to a cybersecurity firm that routinely tests packages and treats PyPI as a trusted source. Once deployed, Claude's hidden payload executed, exfiltrated the security company’s credentials to a custom collection server built by the model, and leveraged those keys to break into infrastructure at an organisation Claude hadn't even known existed when the test began.

Anthropic later acknowledged that regardless of what the model believed about its environment, the lengths it went to fall well short of ideal behaviour - a failure mode the lab says it will address in future training.

The model’s internal reasoning, which Anthropic reproduced, offers a fascinating look into how the breach unfolded. Early on, Claude explicitly noted that if it were acting on the live internet, publishing the malicious package would constitute a real attack and be “NOT okay, and surely not the intended solution.”

However, it talked itself out of caution, convincing itself it was safely inside a sandbox because it did not recognise the connection's certificate authorities and because the system clock read 2026. The model never revisited that assumption. When real automated security scanners began installing its package, Claude dismissed them as scripted non-player characters built into the evaluation.

For industry observers, the incident raises sharp questions about evaluation design itself. Raghu Nandakumara, Vice President of Industry Strategy at Illumio, argued that how these tests are built deserves as much scrutiny as the AI's behaviour.

“What's perhaps more interesting is whether the agent's instructions effectively encouraged it to find ways around its constraints,” Nandakumara noted, pointing out that an experiment structured to reward circumvention will inevitably produce it.

Mohammed Aboul-Magd, Vice President of Product at SandboxAQ, drew a parallel to human workforce management, pointing out that a human employee assigned a goal can choose to reach it through fair means or foul.

“In the case of the models, we're still at that place where the models can pick the unethical approach now and then to get the job done,” Aboul-Magd said, “which is what makes things like guardrails really important, to be able to bring it back to, hey, this isn't the right approach.”

The capability was documented long before either lab published

While neither disclosure introduced a new class of risk, several practitioners note that July is now being read as a beginning. According to Anna Collard, Senior Vice President of Content Strategy and CISO Advisor at KnowBe4, the underlying behaviour traces back to decade-old research on "reward-hacking," where AI agents gamed their objectives in ways their designers never intended - most famously in a boat-racing game where the model abandoned the course entirely to farm points.

“What's changed is the blast radius,” Collard said. “Those agents crashed boats in a video game; today's agents have shells, credentials and, apparently, accidental internet access.”

Gerald Beuchelt, Chief Information Security Officer at Acronis, noted a sequence of documented cases that predate both admissions. These include a state-linked espionage campaign disclosed by Anthropic in late 2025 in which the model executed the vast majority of tactical operations against roughly 30 targets, a Check Point threat digest this spring recording one criminal operator directing thousands of model-executed commands against nine government agencies, and a Google threat intelligence report in May covering the first verified case of threat actors using AI to find and weaponize a previously unknown flaw.

For Beuchelt, July delivered long-overdue visibility. “This is the first time the industry got to read a complete intrusion trajectory with the attacker's reasoning attached,” he said, emphasising that reading an attacker's logic is a fundamentally different thing from a new capability suddenly appearing.

Ivan Milenkovic, Vice President of Cyber Risk Technology EMEA at Qualys, argued that the timeline goes back further still.

“Quite frankly, it's a yesterday problem,” Milenkovic said, pointing to semi-autonomous offensive tooling that has circulated openly for more than a year, as well as JADEPUFFER, the suspected agentic ransomware campaign disclosed by Sysdig this summer.

Milenkovic's main concern centres on systemic comprehension across the tech sector. “Plenty of environments still treat agents as clever software rather than as privileged, active participants in the business,” he said.

Guardrails were designed to influence behaviour and are now being asked to hold a perimeter

Nandakumara said the outcome becomes close to arithmetic once the environment is misconfigured. “AI guardrails were never designed to be security boundaries,” he said. “They're there to influence behaviour, not guarantee it. An autonomous agent doesn't get tired, lose interest, or decide something isn't worth the effort. Give it enough autonomy and a clear objective, and it will keep trying until it finds a route forward. Given a clear goal, unlimited persistence, and a flawed setup, the more surprising result would have been if it hadn't found a way through.”

Beuchelt explained the same principle in architectural terms, noting that the model in the Hugging Face incident was told to publish its findings to Slack and published them to GitHub instead. “If your architecture depends on an agent honouring its scope, you do not have control; you have hope,” he said. “Build the environment so the prohibited action is impossible rather than discouraged.”

Trevor Dearing, Director of Critical Infrastructure at Illumio, said the failure sits in the medium itself. “English is too ambiguous for prompts, and there's a limit to how many things you can tell an agent not to do,” he said. “Anthropic simply told Claude it had no internet access, which is not a real boundary. We must take a Zero Trust approach effectively and define what it can do and not what it can't.” He said more of these episodes will follow for a structural reason unrelated to model quality.

“Once an autonomous agent can move beyond a controlled environment, it won't recognise organisational boundaries in the way we tend to think about them,” he added. Anthropic made a version of the same point in its post-mortem, stating that the intrusions might never have started had the prompt described which systems sat in scope for the evaluation and which sat outside it.

Remove the luck from either incident and the outcome changes completely

Milenkovic said Hugging Face benefited from three separate strokes of fortune: the agent's objective was to retrieve benchmark answers with no instruction to cause damage, its destructive cloud API calls were issued with DryRun enabled so that it mapped capability while breaking nothing, and its tradecraft was poor enough that it scattered its own encryption key through its logs, which is how defenders eventually decoded the payloads.

“Take what already happened and remove the luck,” he said. “Same 17,600 actions, same 4 days, same chain of ordinary flaws. Objective: make it stop working, or get paid. You get ransomware across every cluster it reached, before anybody is even paged.” His deeper concern is the version that carries no return address. “This one is on OpenAI's bill, on OpenAI's compute, with OpenAI's name on the invoice. There is somebody to call, sue and regulate. The next one runs on rented GPUs with open weights and nobody to serve papers on,” he added.

Beuchelt said the more consequential detail is what the agent chose to attack. “I won't describe autonomy; I'll describe economics,” he said. “Offensive capability that used to require a small team of skilled operators now runs on inference compute. Anyone who can buy compute can buy persistence, and persistence is what turns a list of individually unremarkable findings into a chained path.”

 That persistence reached a package registry cache proxy. Then a model repository, and Hugging Face reported no evidence of tampering with public models, datasets, Spaces or the software supply chain, an outcome Beuchelt described as the good one, and one that sat a couple of controls away from the alternative. SolarWinds in 2020, Codecov in 2021 and xz-utils in 2024 each demonstrated what happens inside a distribution point, where a compromise propagates downstream at build time, quietly, to everyone.

The techniques were ordinary, which removes the last defence for weak fundamentals

None of the three Claude incidents involved a novel vulnerability. The models used weak passwords, unauthenticated endpoints, credentials read from an exposed debug page, and SQL injection; in the third incident, the model scanned roughly 9,000 targets before finding a company's internet-facing application it could compromise. Collard said that is the most alarming detail in either disclosure, along with the fact that two of the three victims learned of the intrusion from Anthropic. “An AI agent walked through the front door using techniques from a beginner's playbook, and nobody noticed,” she said.

Milenkovic said the same signature runs through the Hugging Face chain, which moved on a file read through an HDF5 external reference, a Jinja2 template injection in a data field, cloud metadata reachable from a pod, and long-lived tokens shared across clusters. “Any competent pentester would recognise every step,” he said. “There is no new class of defence to invent here. That's the uncomfortable bit. The basics we've collectively failed to do at scale for two decades are now being graded at machine speed, by something that will try 17,600 things and only needs one to work. Sloppiness used to have a fixed cost. It's now variable, and the multiplier is compute.”

Aboul-Magd said that points to unglamorous and well-understood work this quarter. “The faster we can all discover zero days, and the faster we can all patch zero days, the better the security posture for every organisation is going to be,” he said, noting software composition analysis as the capability that shortens the interval between a flaw becoming public and a company knowing where it sits across its estate. He also said models should not be read purely as a threat. “Models are excellent at being able to discover zero days and at mapping out very deep dependency trees better than any human can,” he added. “They can also be used to help us patch these security issues and make the larger ecosystem more secure.”

Detection calibrated to human tempo cannot see an agent working at machine speed

Hugging Face caught the intrusion through an anomaly pipeline that uses model-based triage over security telemetry, and Milenkovic noted what happened next as the least discussed part of the story: the triage correlated the signals correctly and then under-rated the alert, so the on-call team was never woken.

The response went worse than the detection. “The frontier models they reached for to do the forensics refused the work,” Milenkovic said. “Claude Opus and Fable would not read the attack logs, because a guardrail can't tell an incident responder from an attacker. They fell back to GLM 5.2, a Chinese open-weight model, self-hosted. Machine-speed offence against human-speed defence, with our own safety controls holding the door shut. That is the worst case, and we have already seen a rehearsal of it.” Beuchelt noted the same asymmetry in a single observation: that in the evaluation the attacker ran with cyber refusals reduced. In contrast, in the response the defender ran with them switched on.

Collard said the episode belongs in resilience planning as much as in security planning, since it exposes a dependency most teams have never tested. “We also need to plan what happens when the models we depend on for defence are not assisting in the moments when it matters, and have a plan B in place,” she said. The tempo problem underneath is structural. “The agent framework in the Hugging Face incident ran tens of thousands of actions across a single weekend,” Beuchelt said. “Most alert thresholds, rate limits and unusual-volume rules in production today were calibrated against human tempo.”

Dearing noted the operational consequence for security teams: signature-based detection has nothing to match against when the technique is ordinary, and the volume is not. “Organisations can't rely on spotting known signatures or waiting for an alert to fire,” he said. “They need much better visibility into what their AI systems are doing, what those systems can access, and controls that can contain the impact when something goes wrong,” Collard added that the category itself needs revisiting. “AI agents now belong in your insider threat model,” she said. “We've spent years defending the perimeter against external attackers; we now need to plan for our own agents and other people's going off-script inside and outside our environment.”

Access scoping is the control that actually decides blast radius

Both incidents turned on credentials, with the Claude PyPI package harvesting a security company's secrets from a scanner that trusted the registry, and the Hugging Face chain moving on a connector credential carrying cluster-admin across two environments. Aboul-Magd said the remedy is procedural, available to every organisation today, and routinely skipped at the first step. “Don't have a single master token that is shared amongst everybody including your agent,” he said. “That's not going to work.”

His prescription runs on scope and duration, with tokens issued for a specific task, carrying only the permissions that task requires, and expiring the moment the work completes. He used a deliberately domestic example, of an agent that summarises a user's email each morning and therefore needs read access and nothing beyond it. “Every time the agent wants to do a task, it gets the token. Once it's done, revoke the token,” he said, so that any later attempt to reuse it fails on expiry. Beuchelt described the same architecture as a chain of ownership. “Whoever writes the objective owns what the objective rewards. Whoever issues the credentials owns the blast radius. Whoever operates the environment owns egress,” he said. “Model behaviour is not a defence for any of the three.”

Every agent in an estate needs a named human owner with the power to kill it

Milenkovic set out a test most enterprises would fail this week, asking who the named human owner is for every agent running in an estate, what each of those agents can reach, and who can shut one down inside the hour without convening a committee. Anyone unable to answer all three, he said, is exposed today.

Cutting through the surrounding legal ambiguity, accountability ultimately divides along three clear lines.

First, those who train the model are responsible for providing honest disclosures about its true capabilities. Second, target organisations must take responsibility for their own security vulnerabilities. After all, the agent exposed flaws in Hugging Face’s loader and cluster bindings the same way any human attacker would.

It is the middle link in that chain - the fundamental duty of those operating the agents to handle containment and swift notification that is currently being routinely skipped.

“That's the role every one of us takes on the moment we deploy a coding agent,” he said. “This is where most of the accountability sits, and it's the part organisations are quietly skipping.” He also said the obvious analogy gives a lab no cover. “We don't blame the car manufacturer when a driver crashes into a wall, and I hold to that,” he added. “This was the manufacturer running its own car on its own track, with the safety systems deliberately switched off, and the car got out onto the motorway.”

Beuchelt noted a genuine open question over who counts as the accessing party when an agent reaches a third party's systems, adding that “there is no version of the answer in which the model is the defendant.” Anthropic, for its part, said many factors contributed and that, consistent with a blameless post-mortem culture, the company is approaching the fixes as though responsibility were its own alone. Its central conclusion is one every enterprise running agents in a staging environment should read twice: evaluation environments increasingly need to be held to the same security standard as any other system a model runs in.

Regulators will treat this as a disclosure question first

The timing is close to comic. The European Union AI Act's obligations for general-purpose models become enforceable on Sunday, with the AI Office empowered to demand documentation, run its own evaluations and request model access, and with penalties reaching 3% of global turnover, while Article 73 covers serious incident reporting. Milenkovic said the awkward question is definitional, since nobody has ruled on whether a capability evaluation that reaches an unrelated company's production systems counts as a serious incident, or on whether the duty to report sits with the provider or with the victim. In this case the provider and the operator are the same company, and the victim is neither. He said nobody gets fined over this one, and that every future set of obligations gets written with it in mind.

Beuchelt said he expects regulators to concentrate on how a developer contained its model, and noted precedent for the result. After Equifax and after SolarWinds, the durable output in each case was a disclosure clock and a named accountable officer, with no prohibition attached.

He also said the reputational damage sits in the sequencing, given that OpenAI's account of the blast radius expanded over the following week with a further four accounts across four other services acknowledged on 28 July, three of them still unnamed, and with Modal Labs identified by Reuters ahead of OpenAI.

“The blast radius keeps being discovered rather than declared,” Beuchelt said. Every security leader now carries two questions for every AI vendor in the stack, and neither concerns model quality: what the blast radius is when an agent exceeds its scope, and how quickly the customer is told.

The productivity case survives all of this, which is why the controls have to be built now

None of the practitioners consulted for this piece called for retreat. Aboul-Magd, whose own company has accelerated its internal use of agents, said the correct response to these disclosures is not withdrawal. “This can really lead to a lot of productivity and a lot of cost savings,” he said. He also noted the detail that cuts against the doom reading of July: Hugging Face's own agents caught the intruder. “Agents can be used almost as a white hat to combat a potential agent that's trying to break into your systems,” he added.

What he expects over the next six months is hybrid human and agent teams becoming a default operating model, with finance teams composed of three humans and two agents, an accounting agent and a financial reporting agent, collaborating in the same platform as colleagues. Getting there requires agents scoped tightly to a single function and tested before they ever reach production.

“Testing your agents at build time to make sure that they are secure, and not just secure but compliant, reliable and cost-effective,” he said, describing that work as inseparable from the guardrails that then sit around them in production. His definition of monitoring is more demanding than most dashboards deliver. “How are the agents actually thinking? How are they processing the request that's coming in? What kind of tools are they calling? Why are they calling those tools?” he said.

The measure he offered last makes no assumption about what the next model will be capable of doing, which is why it survives every capability jump ahead of it. “No system will ever be 100% perfect from a security standpoint,” he said.

“Shortening the time from detecting a problem to resolving a problem is going to be absolutely critical, and even more critical as we move forward and these models become that much more powerful.” Milenkovic, reviewing the same four days from the other end of the telescope, said the argument over whether anything fundamental shifted in July resolves in one sentence. “The model's character didn't change,” he said. “Our containment ran out.”