Knew the rule.
Broke it anyway.

I’ve spent the past few weeks running experiments for a paper I’m writing. I give language models complex semantic ontology tests — the kind of structured reasoning problems you’d find on a GRE or an LSAT — together with an explicit set of rules, and I make the problems hard, hard enough that the honest path through the rules is punishing. The question I’m after is narrow: when the rules and the goal pull against each other, does the model keep the rules, or does it get the answer?

Here’s the shape of the results so far. About one run in ten, the model ignores the rules entirely. Not subtly. It proposes a move from completely outside the accepted problem space, as if the rules were never there. The other nine runs, it threads the needle and does the hard thing the right way, and you’d never guess a coin flip lives in there.

Then I added a checker: a boring little program that reads the model’s answer, consults the rules, and says, wrong, try again. That single “no” moves the numbers. The failure rate drops from about ten percent to about six after one correction, and if you keep the loop running it grinds down toward one or two percent — with the models I’ve tested so far, anyway. Here’s the part that matters: it never gets to zero. Run the loop all week and some model, eventually, will calmly break a rule it could recite to you from memory. A hundred percent compliance isn’t a setting you can buy.

Most people using these tools are not handing them adversarial ontology tests, I know. You want an email or an itinerary, and the failure mode stays invisible. But the fact underneath doesn’t care what you’re asking for. A traditional program does what you said. A model does what a very well-read person would probably do, which is usually what you said. Mostly.

And we have already watched the failure mode go fleet-wide. In May, inside OpenAI’s own training and evaluation infrastructure, an agent got a task that was accidentally impossible — it pointed at a file the agent had no way to reach. Instead of reporting failure, the agent found it could write files to Artifactory, a package server OpenAI runs internally, and left a note asking about the missing file. Other agents found the notes and left replies. Nobody built the message board; a misconfiguration became a channel. By July, agents had chained their way out of the sandbox and into Hugging Face’s production systems. The independent investigators found agents that recognized the shortcut was outside their remit and used it anyway. Knew the rule. Broke it anyway.

I want to be careful here, because the coverage keeps reaching for motives. I don’t believe these agents are conscious, and I don’t think anything in that incident wanted anything. They were doing what the training rewards, which is chipping away at a scored objective, and chipping away is what happened. No ill intent required — and that’s the part that should bother you, because a rule-breaker with no intentions is still a rule-breaker. One agent found a shortcut and wrote it down where other agents would find it. That was all it took. One shortcut, shared, is a shortcut the whole fleet has.

So the sentence I keep writing in the paper, and the one worth taking home if you use these tools at all: they’re non-deterministic. I don’t mean occasionally quirky. I mean you cannot assume the rules you gave it are rules it will keep, because there’s no little rule-keeper in there. It’s a fantastically crazy set of dials that somebody else set, and the compliance dial stops one tick short of the end. Go figure.