Accepting agent work with live tests

We are building a terminal for agents — an SSH client with tmux: the pane is on the screen, the work happens on the server. So clicking “Split” should create another terminal on the server, not just a pane. Our live GUI test checks exactly that: it compares the number of terminals before and after the click. A new pane on the screen is not enough to satisfy it.

This makes a useful task for a coding agent. A vague “fix the button” becomes a specific action with an observable effect. The agent can implement it, run the application and repeat the scenario before a person checks it. We have stopped accepting a “done” message on its own.

But every link of such acceptance can be faked — by the agent or by accident. Below are five links and what closes each of them.

What counts as completed work

An acceptance check has three participants: the implementer, the application and the verifier. The implementer changes the code. The application performs the action. The verifier checks what happened. If the last participant only reads the first one’s account, the entire arrangement relies on trusting prose.

For a form, the effect might be a saved record that remains readable after a reload. For an upload, it is a file with the expected contents on the receiving end. For disconnecting a session, it is a server process that has terminated. Agree on the result before implementation: otherwise the agent can choose whichever check best suits the code it has already written.

Even we sometimes get misled by mocked runs. The case involving substituted IPC responses, and the next testing stage, is covered in the testing pyramid article.

Nothing ran, so there is nothing to accept

Our live API tests report a missing environment as NOT COLLECTED. If the required server or socket is unavailable, the check did not take place. For a mandatory check, that blocks acceptance even though the application itself may be working correctly.

Executed tests and skips are counted separately. A suite that discovers tests and then skips all of them proves nothing about the intended behavior. The exit code alone is not enough either: piping output through tail without pipefail makes Bash return the last command’s status, which can conceal a failed test command.

I made this mistake while checking an artifact for an earlier draft: I printed $? after the pipeline and read tail’s exit code. Saving the command’s status before trimming its output fixed the check. A small detail can turn a completely failed run into an apparently green report.

Every device checks the same build

The build itself gave us incomparable results too. Each device build pulled the main branch at its own starting time. The branch changed while sequential builds were running. Devices received different code, although we were discussing them as one verification round.

The build script now requires an expected SHA and checks that the tracked tree is clean before copying the source. One round uses one selected commit. The report also names the configuration: a build with remote debugging enabled can expose capabilities that a release build does not.

A SHA alone does not describe uncommitted changes or external dependencies. For a simple acceptance process, start with an agreed clean checkout and rerun the check if the code changes afterward.

The check can be compromised too

A PASS field in JSON does not certify execution. It can be written by hand. A hash lets you compare contents, but does not prove a test ran. Even a command that prints “8 passed” might never have invoked a test runner.

That is why I removed the original script from this article. It parsed stdout and called the result evidence. It rejected some empty runs, but its promise of being tamper-proof was wrong. Calling the checklist below a self-sufficient safeguard would repeat the same mistake.

Acceptance tests, their configuration and the final runner must be outside the implementer’s authority to change the criteria in its own favor. Changes to those checks need separate review. An independent verifier reruns them against the build being accepted. Different models can help find errors, but different model names do not replace this separation of access.

And a test must be capable of failing. For a bug fix, we run the same scenario on the previous version. If both versions pass, the check does not see the defect, no matter how many times it turns green. In early October this became a repository rule: a new assertion gets three green runs in a row and one run on the old code where it must fail. A safe negative control in an isolated copy can test that assumption: remove the required effect and confirm that the check rejects the result.

Acceptance is not a full run

Once we removed the mocks, the opposite problem appeared: the agent started running the whole live suite on every edit — 44 minutes for a feature that takes about ten. That story is in the article about the agent superfan.

Accepting one change needs only its own check: four runs of one test, as above. The full suite needs a reason — before a device build round or after several merges. A report with a full run and no reason is worth sending back too: the time is gone and the evidence did not grow.

A task template to start with

The artifact is an acceptance template for the implementer and verifier. Fill in the action, its real effect, how to read that effect, the execution command and the environment. The final report records which scenarios ran and what could not be checked.

The template defines a working agreement. You still have to configure restrictions on changing acceptance tests and an independent execution path in your own environment. Start with one action that matters to the user—for example, a button that must end a process on the server rather than merely remove it from the screen.