When an AI Agent Says Done, What Has It Actually Checked?

By Jhony Vidal
Published 15 November 2026
Updated 15 November 2026
3 min read

For: Product owners and engineers deciding how much work to delegate to AI agents

Define completion with observable evidence, and test how an agent detects and recovers from mistakes.

On this page

An assistant says it has updated a customer record and sent a confirmation. The message reads well. But the record belongs to the wrong customer, and the email went out before anyone noticed.

This is a fictional example, but it exposes a real problem with evaluating agents: a convincing final message tells us very little about the work that produced it. Even checking that an email was sent would miss the wrong recipient.

As we plan work for 2027, I think one useful question is how to make completion verifiable. An agent that takes several actions needs checks along the way, a way to recover and a clear account of what remains uncertain.

What the 2026 research adds

Towards a Science of AI Agent Reliability, first published in February 2026 and accepted at ICML 2026, separates reliability into consistency, robustness, predictability and safety. Its evaluation of 15 models on two benchmarks found that capability gains brought only small reliability improvements.

That gives teams a useful distinction: succeeding more often does not necessarily mean failing in ways we can anticipate or contain. The study covers a limited set of tasks and agent implementations, and some safety assessment uses model judges. Its results should prompt local testing, rather than become a failure-rate estimate for your product.

A September 2026 preprint, Locating Hidden Failures Makes Long-Horizon Agents More Reliable, examines sequences of agent actions across software engineering, computer use and science. It reports that a run can look correct at the end while earlier mistakes remain unresolved. It also documents harmful actions within runs scored as solved.

The paper’s failure-location tests are especially relevant to review tools. On its software-engineering and computer-use tasks, none of the six tested frontier judges pinpointed the first mistake in more than a third of incorrect runs. That is a result for those tasks, models and exact-step scoring rules, not evidence that automated review is useless. It does mean that asking another model “was this correct?” is not a complete verification plan. The preprint is recent evidence, not settled guidance for every workflow.

My practical reading of these papers is to inspect what happened between the request and the final message. The examples below are design proposals, not interventions these papers proved will work in every setting.

Define done before the agent starts

For the customer-record example, “done” should mean more than a successful tool response.

StageEvidence neededIf it is missing
Find the customerA unique record matching the approved identifierAsk for clarification
Prepare the updateExact fields and values ready for reviewKeep it as a draft
Apply the changeRead back the saved values and record versionReport an uncertain or failed update
Send confirmationApproved recipient and a receipt from the sending serviceCheck status before retrying

A receipt proves what the service accepted. It does not prove the person read the message. That distinction belongs in the completion report.

The product owner defines the promised outcome and acceptable exceptions. The engineer turns them into observable checks. A model may help interpret a response, but a plain comparison of saved fields is preferable when the result has an exact answer.

Find the first wrong turn

Suppose the final record is wrong. Reviewing only the last write may reveal valid field names and a successful response. The mistake happened earlier, when the agent selected the wrong customer.

Keep enough evidence to reconstruct that decision: which record was selected, which identifier justified it, which values were approved and which operation was submitted. Record versions and tool errors matter too. Avoid retaining private message bodies when identifiers and redacted fields are sufficient.

During review, mark the earliest decision that lacked support. Then ask what would have detected it before the next action. This produces a more useful fix than adding “be careful” to the agent’s instructions.

Practise recovery, not just completion

A workflow may handle the normal path and still fail when a tool times out after accepting a change. Retrying immediately can repeat the effect.

In a test environment, interrupt the process at that boundary. Can it look up the original operation using a stable identifier? Can it continue from confirmed state? If it cannot establish what happened, does it stop and tell the person what is uncertain?

Recovery is not always undoing an action. A sent email cannot be unsent by restoring a database checkpoint. Some failures need a corrective action and an explicit handover. Decide who owns that response before automating the step.

Run the same task more than once

A single successful demonstration says little about consistency. Try a small set of representative tasks repeatedly, then change one condition: reword the request, remove a required field, return a stale record or make a tool unavailable.

Record completion, incorrect actions, successful recoveries, unnecessary handovers and time spent by the reviewer. Separate serious boundary failures from ordinary task errors. One unauthorised send should remain visible beside ninety-nine correct drafts.

A useful comparison might be between a fixed workflow and an agent that chooses its own sequence. If both solve the same cases, compare review effort and recovery behaviour before adding more freedom.

Give the reader evidence in the final message

Instead of “Everything is done”, a useful completion report might say:

Updated the delivery preference on customer record C-1042 and checked the saved value. The confirmation service accepted message M-318 for the approved recipient. Delivery to the recipient has not yet been confirmed.

Those identifiers are fictional. The important feature is the separation between what was checked and what is still unknown. A person should be able to open the underlying record when their permissions allow it.

For a first exercise, choose one task your team wants to delegate. Write down the evidence required at each action, introduce one failure and observe the recovery. That will reveal more about the next engineering decision than another polished demonstration of the happy path.

Put the idea to work

Choose a small local project to inspect a real result, or find your next reading route.


Tags

ai-engineeringevaluation

Share

The Author

Jhony Vidal

Jhony Vidal

Lead AI Engineer