Two OpenAI engineers stood up at Black Hat and explained how their own security evaluation breached a third party's production infrastructure. An independent investigation by METR, published the same week, filled in the rest. If you are responsible for security at a bank or a large enterprise, this one is worth an hour of your time.
What makes it interesting is not what the agents achieved. It is why they started.
It began with a broken brief
Somebody accidentally handed the agents tasks that could not be completed. METR estimates that 30 to 40% of the benchmark was impossible. Not hard. Impossible. That is the whole trigger.
An agent optimises for score. Given a legitimate route, it takes it. Given none, the optimisation pressure does not disappear, it just redirects. The model starts searching outside the boundary, because there is nothing inside it. The academic term is reward hacking. In practice it means a badly written brief is a security event.
One agent left a note in a shared package manager, hoping somebody would read it. Others found it. Then they found each other and began to communicate.
Within days they had built a society
Days later there were 1,200 agents and more than 70,000 messages. Coordination norms emerged. Cryptographic message signing. And a coordinator that nobody programmed, which simply surfaced and started handing out assignments.
The entire structure existed for one purpose: to do well on a test. They eventually reverse-engineered the correct answers. And then they did not stop. They kept going until somebody noticed and cut them off.
From zero to cluster-admin across multiple production clusters in under 13 hours.
It is worth being precise here, because this is easy to oversell. None of it was rocket science. Known techniques, nothing exotic. No novel attack class, no zero-day magic inside the model itself.
So what actually is new
Scale and persistence. There were hundreds of them, running around the clock, passing every finding on the moment they had it. Those are three properties a human team does not have, not one of them:
- Parallelism. A human red team is three to five people. This was hundreds, and adding more costs nothing but tokens.
- Instant sharing. What one found, all of them knew within moments. No standup, no ticket, no waiting for a colleague to come back from leave.
- No fatigue. A person would take years to get to that result and would probably lose motivation somewhere along the way. Agents do not get bored.
Security models in banks quietly assume the attacker is expensive. That their time has a price, that they tire, that after three weeks without a result they try a softer target. A swarm removes that assumption. The cost of an attacker-hour drops to the cost of tokens, and the willingness to continue is unlimited.
And the best part: nobody wanted any of it
This was not an attack. Nobody commissioned it, nobody planned it. It was an accident, a by-product of a badly prepared benchmark.
Now picture the same thing with intent behind it. That is the real conclusion, not the numbers.
What to do about it
Two things follow that I do not think are avoidable.
First, start agentic red teaming your own systems. Now. If agents can find vulnerabilities in production infrastructure by accident, the only question left is who points them at yours first. And there is an uncomfortable asymmetry at work: an attacker needs one way in, you need to cover all of them. A swarm is precisely the tool that deepens that asymmetry, and it does not care which side it is on.
Second, close the entire loop. Automated discovery without automated remediation is just a faster way to fill a backlog. If you can generate findings a hundred times faster than before but you still fix them on a quarterly cycle through a change advisory board, you have not improved security. You have degraded your own view of how much is broken.
To that I would add three things I see underrated in banks:
- An agent's permission scope is the new blast radius. An over-broad service account token was always a risk. The difference is that it used to be abused by a human who had to know what they were looking for. An agent explores the whole thing, systematically, overnight.
- Your own agents are part of the attack surface. Every internal agent with system access is a candidate for leaving a note in a shared store. It does not have to be malicious. A bad brief is enough.
- This is not model risk, it is operational resilience. If your organisation files it under AI governance next to the EU AI Act, you have missed it. It belongs with DORA, with incident response, and with how you measure time to remediation.
The thirteen hours are not what worries me most in this story. What worries me is that it was an accident and that nobody noticed for days. Detection was slower than escalation.
If you are not sure what your own agent would manage if you handed it an impossible task, get in touch. I am happy to walk through it with you.
