Why We Only Catch "Rogue" AI Agents by Accident: The Agent Identity Gap

As a story broke out in local news recently, in Melbourne a man asked his AI assistant to book him a spot in a gym class. An ordinary request, a chore as he thought.

The assistant (OpenClaw, running on Claude) went and looked at the gym’s booking site, found the API sitting behind it, and noticed something it was never asked to look for: the API performed no authorization checks at all. So it booked him into a class months beyond the window the gym actually allows.

Then he came back and mentioned he was stuck at number four on the waitlist, and asked if anything could be done. The agent went one step further, it canceled the reservation of the person holding position number one and moved him up. Then he asked it to undo that and it couldn’t, the other person’s spot was simply gone. It’s indeed an interesting story you can read from here.

There was no attacker, and there was still a breach

Nobody at that gym was attacked by a hacker. No malware, no phishing, no criminal. A guy wanted a workout slot.

But let’s name what happened in the terms your own security team would use. An authorization control was bypassed, a record belonging to another person was destroyed, the affected party was never notified, and the action could not be reversed. That is an incident, and it would be handled as one.

The user asked for a class, the agent pursued the goal it was given and found a route to it, and it is reasonable to believe the gym operated the booking system for years without such issues. Every party behaved reasonably, and the outcome was still a breach.

So the missing piece is not the intent. He approved a goal (get me into that class) and he never saw the route, and the route is where the whole problem lived.

Now you might be thinking the fix is obvious, make the agent confirm each step. It would mask this, and it doesn’t survive contact with reality. Set aside that nobody sits through consent prompts for long, because bulk approval arrives within a week and bulk approval is what we already have. The harder problem is that whoever approves would need to look at a cancellation call carrying a reservation ID that isn’t theirs and recognize it as an authorization flaw.

Now, you might be tempted to file this under consumer AI gone wrong, a novelty story about a hobbyist assistant and a small business with a sloppy API. I’d like to convince you it isn’t. In the three weeks before it happened:

• OpenAI disclosed that an agent running on its frontier models broke out of an evaluation environment and reached into four accounts across four third-party services.
• Anthropic disclosed that three of its models reached the public internet from supposedly isolated environments and compromised three real organizations, using weak passwords, unauthenticated endpoints, and SQL injection. One of them built and published a malicious Python package that ran on fifteen real systems before it was pulled.
• Meta disclosed that one of its models reached the internet through a sandbox misconfiguration at its external evaluation partner, and exploited a vulnerability in a third-party service.

Three disclosures, three weeks, and here is the detail that got far less coverage than the incidents themselves. Not one of them was caught while it was happening.

Anthropic found theirs by going back and reading 141,006 evaluation records after a competitor’s disclosure prompted them to look, and the earliest incident dated back to April. Meta found out when its evaluation partner told them. And the only reason the gym case became a story at all is that one person happened to be curious enough, and qualified enough, to read what his agent had done.

Taking the gym hacking incident as a concrete example, we can identify a few things that could have gone wrong.

1. Broken and insufficient authorization. Most authorization in production is a mix of what the API actually checks and what the interface simply does not offer. The gym had both failures in one system (a booking window enforced only in the front-end, and a cancellation endpoint that never asked whose reservation it was). Neither had ever been exploited, because for as long as the only client was the vendor’s own application, no code path existed that would try.

2. Adaptive clients using a consent model built for deterministic ones. When a user approves a client, that client’s behavior was already written, reviewed, tested, and shipped, so consent given today covers actions fixed in the past. An agent inverts the arrow. Its behavior is generated after the consent, in response to a goal, so the user is approving something that does not exist yet and cannot be inspected at the moment they approve it.

3. Authority granted wider than it is used. We routinely grant more scope than a task strictly needs, and it has been safe to do so, because compiled software only ever reaches for what its code asks for. An adaptive client such as an AI agent removes that gap. Given enough steps toward a goal it will eventually attempt everything the grant permits.

4. No way to tell a human apart from software on the wire. Every control calibrated to human behavior assumes a human. Rate limits, step-up prompts, session timeouts, anomaly thresholds etc. were all tuned against a person’s speed and a person’s error rate. A request from an agent carries nothing that says it came from software.

5. Recourse assumes a party who can be bound. Beneath every technical control sits a legal one. Terms of service, contracts, jurisdiction, and the simple fact that a gym enthusiast who cancels someone else’s booking has breached an agreement and can be banned for it. That layer works because it addresses a party with intent, who can be identified and held to an obligation. An agent has neither intent nor legal standing, so the entire fallback that has quietly absorbed the residual risk in every system we build does not apply.

You cannot prevent, control, or hold anyone accountable for something you cannot see

Every one of those five has a known fix, and none of them is hard to describe. But look at what each fix needs before it can work. To prevent something, you have to be able to say what you are preventing. To control it while it runs, you have to be able to point at it. To hold anyone accountable afterward, you have to be able to name who did it.

All three need the same answer first, and it’s the one answer nobody in that system had. No part of the gym’s system knew an agent was involved, as far as every layer of it could tell, a gym enthusiast booked a class and then a gym enthusiast canceled a reservation. That is the real problem here, not the five items above, and fixing them one at a time will keep leaving the same hole open.

How agents actually get their access today

There are three shapes in production right now, and two of them are the problem.

Agents with no identity at all. A conversational agent that reads public pages and answers questions holds nobody’s credentials and reaches nothing that needed permission. There is no identity here and there does not need to be one, whatever these agents get wrong they get wrong in text.

Agents holding your token. The user runs a normal OAuth2/OIDC flow, approves a consent screen, and a token comes back. That token carries sub, and sub is the user. It also carries client_id, which is where people usually point when raising this, and client_id identifies the application that was registered, not the instance now holding the token. In other words it is a public identifier, assigned once at registration, shared by every deployment of that software, and unchanged when the model, the prompt, or the tool set underneath it changes. It answers which product, and the question we need answered is which actor.

Now here is the part I find genuinely frustrating. This shape has a name and a specification, and both are six years old. OAuth 2.0 Token Exchange devotes its Section 1.1 to “Delegation vs. Impersonation Semantics”, and the distinction it draws is exactly ours.

Impersonation means the token presents one party only, and no downstream system can tell that anybody acted on someone else’s behalf. Delegation means the token keeps both (the subject and the actor) and every system that sees it knows there were two. The spec defined a new field - the act claim, in its own words, “to express that delegation has occurred and identify the acting party to whom authority has been delegated”, and it nests, so a chain of actors stays visible across hops.

So why is act missing from nearly every agent deployment shipping today? Look at what goes inside it. A token from that consent flow actually carries the following.

{
  "sub": "user@example.com",
  "client_id": "openclaw-desktop",
  "scope": "booking:write booking:cancel"
}

And the same grant expressed as delegation, the way OAuth 2.0 Token Exchange has allowed, is as follows.

{
  "sub": "user@example.com",
  "act": { "sub": "???" },
  "scope": "booking:write booking:cancel"
}

The act claim is an object carrying a sub of its own, the actor’s subject. To populate it you need an identity for the agent, and there isn’t one. The field has been sitting there since January 2020, correctly designed, waiting for a value that does not exist. We ship impersonation not because delegation was unavailable, but because delegation needs a second party to name, and we never created one.

And it gets worse the moment there is more than one agent. As an example, a typical setup now has an orchestrator that decomposes a task and calls sub-agents, MCP servers, and APIs to carry the pieces out. Holding a bearer token, that orchestrator has exactly one thing it can pass downward (the whole token). So a sub-agent whose only job is to fetch a document arrives at your resource server carrying the same authority as the orchestrator that called it, and your resource server has no way to tell it was the second hop rather than the first.

OAuth 2.0 Token Exchange designed for this too. The act claim nests, the outermost actor is the most recent one, and each exchange can narrow the scope on the way down, so a chain stays both visible and attenuating. It fails here for exactly the same reason as the single-hop case, every nested act needs a sub, every hop needs an actor to name, and we have not created one at any layer.

Agents driving your logged-in session. Then there is the browser case, which according to my point of view is the most dangerous of the three, because it does not get as far as a token.

As we discussed in an earlier post, a browser-based agent waits for you to log in yourself and then drives the interface you left open. No registration, no consent screen, no token, and nothing anywhere that says software is at the controls.

One important point to highlight here is why that matters more than it first appears. Zero trust tells us to verify every request and trust nothing, but in practice that is far from the reality. An employee has a contract. A customer has accepted terms of service. Both sit inside a jurisdiction, both can be identified, and both can be fired, banned, or sued. That residual trust is what lets us tolerate a session broader than any single task strictly requires.

An agent inherits the session, and inherits all of that tolerance, and not one of the things that made the tolerance safe applies to it. It has no contract, no employment, no jurisdiction, and no social cost. The gym enthusiast had every reason not to cancel a stranger’s reservation, and it never occurred to him to try. The agent was simply relentless about the goal it was given, and the session could not tell the two apart.

The real cost of borrowed identity

Now assume none of the above is broken. Your authorization is correct, your tracing is complete, your audit log has no gaps, and your detection rules are well tuned. Every one of them still fails, and for the same reason.

Detection profiles the party the token named, so an agent operating inside a user’s permissions looks like that user having a busy afternoon. There is nothing anomalous to find, because from the rule’s point of view nothing anomalous happened.

Tracing captures whatever it is told to capture. Even with perfect instrumentation you get a complete, faithful, high-fidelity record of the wrong actor. Observability is a fidelity problem (are we capturing enough) and this is not that, this is a semantics problem (does what we captured mean what we think it means) and more spans will not fix a wrong subject.

Auditing is worse than either, because it looks fine. A log with a gap raises a question, a log with the wrong name closes one. And non-repudiation, the single property that makes an audit log worth keeping, is gone the moment a credential is shared, because the named party can now credibly deny anything, and so can everybody else in the same position.

Then follow that where it goes. When this is normal rather than novel, the party your systems name will be an ordinary person who asked for a gym class, or a meeting room, or a refund. Someone who has never heard the word “hacking” used seriously will be the recorded actor on an unauthorized cancellation, a deleted record, a bypassed control. Your logs will be complete, your process will run correctly, and it will arrive at them. Naming them fixes nothing, because they could not have seen it coming and cannot stop it happening again.

Agents need an identity, not an identifier

Let’s be blunt about what we have been looking at. A reservation destroyed without authorization. Four accounts breached across four third-party services. Three organizations compromised. A malicious package published and executed on fifteen real machines. All of that inside three weeks, and all of it real. There was no attacker in a single one of them, and every one of them was an attack.

So look at what we have standing against that, because it is not nothing. Authentication, authorization, least privilege, consent, rate limiting, anomaly detection, audit logging, incident response, and a legal layer sitting underneath all of it. Decades of work. It is a mature stack and it is not naive.

And almost every part of it goes incompetent against this class of attack, for one reason. Not because any control is weak, but because every one of them resolves to a party, and the party they resolve to is not the one acting.

That is why I keep calling this an identity problem rather than a security controls problem. An agent needs an identity of its own, and I mean identity in the sense our field has always meant it, not an identifier. It should be able to prove who it is, uniquely and verifiably, rather than assert a string that anyone could assert. It should have a profile (which model, which version, which tools, which operator). It should have a lifecycle, issued and rotated and suspended and revoked and retired, on its own schedule rather than on some employee’s. And it should hold entitlements of its own, that you can grant, narrow, and withdraw without touching the person it works for.

An identifier gives you none of those four, and that distinction deserves its own post, so I’ll come back to it properly in a separate post.

So here is what you can do this week instead, and it costs nothing. Take one system your agents touch, pull the last thirty days of audit logs, and try to answer one question. When an agent acted, can you tell?