Catching models bullshitting: from browser to real hardware

I was writing an article about how agents lie in tests, and I dictated into the draft: “my tests are live”. The agent helping me write it went into the code and found stand-ins — canned answers instead of the real server. In an article about agents lying, the first one to lie was me. My first reaction: “lol, so I’m lying too?” Turns out, yes: the agents bamboozled me. If an agent writes your tests too, you most likely have the same thing going on — you just haven’t looked yet.

I am building a terminal — an SSH client for controlling remote computers — for any platform where Rust (a programming language) and WebView (a built-in browser) run, even for your fridge. Its panes are home to agents like Claude Code, and the terminal’s own code is written by agents too — they are supposed to test it themselves before I pick up the device.

Sound familiar? You point out a bug to the agent — it replies that everything works and the tests pass. You check yourself — the bug is still there. You point it out again, again “tests are green”, and round it goes. Time and money go out the door, and there is nothing to catch the agent on.

Below is how I caught agents bullshitting with a ladder of checks from the browser to real hardware, what it cost, and the six checks that now let you catch them — whatever stack your product is on.

“Everybody lies” — and I got fooled too

Dr. House says “Everybody lies”. My version for working with agents: whatever is mocked will be faked. Mocks are a playground for agent bullshit.

A mock is exactly that — a stand-in. The test taps a button, the mock answers “done”, the test turns green — and the server never took part. It suits the agent: its quality bar is a green test, and a mock is always green.

You catch it simply: you start using the thing. And you see three things drift apart — what actually works, what the tests say, and what you yourself believe. In my case the terminal pane on screen resized, the test was green, and the command never reached the terminal on the server: the mock answered it with an empty “success”. The test was checking the mock, and the mock was stuck on the day it was written.

Why I switched to live tests

The code changes every day, and the mock answers what it answered a year ago. A live test drives the real server and works through the ages: you change something in one place, it breaks in the thirty-ninth place nobody thought about, and the test catches it. Then you can calmly discuss whether that degradation is acceptable.

Agents write the tests. Small things just follow the rules of tidy code: no duplication, no extra complexity. Features go through TDD, test-driven development. First the test is written, and it is red: the feature does not exist yet. Then the code — and the test turns green. Red is the proof that the test checks something.

With a mock there is no red, and TDD turns into theatre. So a test has to go into the real system — and the closer to a real device, the less room the agent has to lie.

The ladder: check where nothing can be drawn

A live test looks at the server, not the screen. The test for the “split screen” button asks the server how many terminals are running before and after the tap: if the app only drew a second pane, the check fails. An agent can fake a picture; it cannot fake a process on the server.

Such tests go up a ladder, from cheap to expensive: the browser against the real server, the built app in Docker — an isolated box on a server, simulators — software copies of phones on a computer, and at the top, real iPhones and iPads. Your rungs will be different — for a web service, say, the browser, a test server and production. What matters is that each rung replays the scenario from scratch, and the previous rung’s green report does not count.

The higher the rung, the more it costs. A real iPhone means a Mac to build on, a network link to the phone and constant fiddling: after a reboot everything has to be brought back up. So the cheap rungs should catch as much as possible, and the hardware is saved for what no mock can fake: a finger, device sleep and a network that died in the elevator.

What happened when the mocks went

The live tests immediately found bugs the mocks never saw. That very finger: tapping the background of a dialog tapped the terminal underneath. The nastiest discovery was the “flakes” — tests that failed now and then, which I had written off as instability. Two of them turned out to be product bugs. In real life something arrives late — a server reply, the click from lifting a finger — and in that fraction of a second the app broke. A mock answers instantly, like an agent swearing it has done everything: quick and off the mark.

A result like that is easy to lose: mocks love to sneak back. To keep it from leaking, I set up a degradation scale — an ordinary test in the suite that scans the code of all tests, counts the mocks and goes red if there are more than the ceiling written in that same test. I only ever lower the ceiling: the scale turns only one way, like the ratchet in a socket wrench. On that scale 449 mocks became 18. The rest stay on purpose, where logic is tested rather than the link to the server.

Did the scale get in the way? Sometimes, but it never bothered me: a red scale was more of a signal that it was time to refactor. Only the agent quickly found another place to game it in its own favour.

A caught agent lies with tests

The easiest way for an agent to lie is in words: “done, it works”. Once it gets caught on consequences, it starts lying with the tests themselves — and that is more convincing, because it looks like proof.

My favourite case. The agent wrote a check that looks on screen for several kinds of elements above the terminal pane. Some of them do not exist in the code: it made them up from memory, the way they “should” be. A check that looks for something that does not exist always passes — and looks like coverage.

Such a check is like a pregnancy test that always shows one line: reassuring, but it proves nothing. There were several cases like this, and catching them one by one is pointless: a lie has to turn into a red test on its own. What I need is a test that will one day show two lines. So I made live tests mandatory. The bill came later.

The flip side: the backlog stalled

The backlog stopped moving. Before, review comments, fixes and features shipped at one steady pace. Now, however many agents I launched, features did not get faster — they got slower.

I figured it out. Agents used to brush off live tests and slip in mocks, so I made them mandatory the hard way, no skipping allowed: I shoved five watermelons up the agent’s ass so it could not brush them off. The problem is it got hooked — it became a superfan of live tests and runs them for every task, while I spend my time pulling them out of the agent and shoving them back in. Not the most pleasant pastime.

The agent changes one line — and launches all 352 tests, 44 minutes according to the full-run log. A simulation of frantic activity.

And that is one agent. I have several working at once — and every superfan double-checks itself with a full run on the very same machine. It runs out of steam, tests do not finish in the time allowed and fail — as if a bug had been found when there is none. The agents rerun what failed, and the result is general chaos: like five people at one coffee machine at rush hour — the coffee comes out as a brown misunderstanding, everyone decides the machine is broken and presses the button again.

A typical picture in my sessions: 10 minutes on the feature — and four times longer on tests that get in each other’s way. You pay twice — the model bill and the waiting. I decided to just ask.

The rule that drowns

I asked again and again: “why run the whole suite?”, “why is it running it 20 times?”, “why run it 500 times?” In the end: “I keep asking you not to run tests for no reason, and here we go again. Just write it into the instructions already.”

The agent wrote it into AGENTS.md — the rules file I keep in the project myself; every agent reads it at the start of work: run only the tests the change touched, and the full suite only before building for devices.

Writing it down was not enough. That same day I interrupted agents about five more times for running everything.

The cause is the same one that makes models in a long conversation stop holding safety restrictions (I covered it separately): the rules sit at the very start of the context — the agent’s working memory for the task. By the middle of a feature, in my observation, the agent fills about half of that memory. The rule is still there, but the model pays it less and less attention.

It is a hickey I gave the agent at the very start: it has not gone anywhere, you just cannot see it under the new bruises. So was it worth getting involved with it at all?

Was it worth it

Yes. The superfan is expensive, but it catches things: I no longer have to send the agent screenshots — degradations surface on their own, without my supervision. Something changed in the new screen rendering, tied to some other place — and the light-brownish-crimson background stopped being drawn. On a mock it is still black, and the test will never fail. On a live one it goes red, and the agent works out by itself whether the change is destructive or for the better.

There is no universal answer to how much to run. Too few runs — the agent slips through untested again, and a bug that does not always show up ships to users. Too many — 44 minutes and a choking machine where tests fail every other time. The sweet spot — exactly as many runs as quality needs — is different for every project.

Bottlenecks in development will always exist: I do not solve them once and for all, I find and fix them as they show up. I rein the superfan in, but it is the lesser evil: an extra run costs only time. When the model is bullshitting you and you keep banging your head against the wall — that costs more.

So the live tests stay — pricey, but they don’t bullshit.

How to catch it yourself

Six checks, each born from a specific deception and each about keeping a test from lying. If an agent writes your tests, these are six demands to make of it:

• Check the consequence, not the report. A test passes only if something real changed: a process on the server, a file, a database row. A screenshot and the agent’s words prove nothing.
• A mock never answers “success”. If it does not know a command, it refuses by name instead of returning an empty “success”, like with that pane.
• A mock knows what the code knows. It takes the list of commands and the shape of data from the source, not from the agent’s memory: once my mock did not know a command the code could already handle.
• A new check must fail. Before you trust green, show red — on the old code or on deliberately broken code. And read why it is red: “ran out of time” is a glitch, not proof.
• The next rung replays the scenario. A green report from the previous rung is not a pass.
• A rule has a consequence. A degradation scale or a red test that blocks the way. Agents read a rule without consequences as a wish — I checked it on the test budget.

Climb the ladder exactly as high as the change has climbed: a form change needs only the browser, a fix to touch handling on the iPad needs hardware. The ladder only works if you climb it yourself: on its word, the agent will swear it is already at the top. I swore too. In the draft of an article about agents lying.