Skip to content
← Journal

Building coding agents that can prove the work

A practical evidence model for trustworthy coding agents.

Start with the claim you need to support

A coding task is not finished because the agent has written a convincing summary. A file may be missing, a test may have failed, or the changed program may not match the request. Verification starts with the behavior the operator actually asked for.

For a bug fix, that might mean reproducing a failure and checking the repaired path. For a UI change, it can include a build, interaction checks and screenshots at relevant widths. The evidence should be proportional to the change, and the final report should distinguish what was checked from what remains untested.

Evidence does not replace permission

A thoroughly tested change can still be outside the request. Verification asks whether something works. Approval asks whether the operator authorizes it. Deleting data, publishing a change or using a write-capable service may require a separate decision even after local checks pass.

The useful final report is concrete: what changed, which checks ran, their results and the remaining limitations. Confidence can accompany that report, but it cannot substitute for any of those details.

Illustrative completion report
An unsupported leap

Everything is fixed and production-ready.

A bounded conclusion

The changed path passes the named check. The deployment environment and unrelated paths were not tested.

Begin with a check that can reject the change

Before changing code, translate the request into observable behavior. ‘Fix navigation’ is too broad to verify. ‘Recover a valid session after client-side navigation without redirecting a signed-in user to login’ identifies a behavior that a focused test could exercise. The acceptance criterion should be about the user’s requested outcome, not about whether the agent managed to edit a file.

A useful check should fail under the broken behavior. If it passes before and after the change without exercising the relevant path, it provides little evidence about the fix. Existing tests can still be valuable: they may protect nearby behavior. Explain that role separately from the regression check so the handoff does not turn a general passing suite into a specific proof it cannot supply.

The example here is illustrative, not a published successful authentication repair. It shows how to make a task small enough to challenge. A real report needs the actual reproduction, repository state and executed check.

A map of responsibilitiesFrom a proposal to a checked outcome
  1. Describe the behavior

    Write an acceptance criterion tied to the user’s request.

  2. Exercise the failure

    Choose a check that reaches the path the change is meant to repair.

  3. Review the evidence

    Inspect the patch and the actual check output, including uncertainty.

A failed check can return the task to investigation.

Read the failure before scoring the model

A test process can fail before it tests anything useful. Collection errors, missing fixtures, inaccessible services and crashes are different from an assertion rejecting a model’s patch. The distinction matters both to the person accepting a change and to the researcher comparing models.

The repository’s evaluation-validity tests include records with unavailable infrastructure, verifier collection faults and model failures. The classifier’s purpose is to stop those outcomes from being pooled as if they all measured the same thing. A failed executed assertion can be admissible evidence about the attempted solution; an environment that prevented the model from participating cannot establish that solution’s quality.

Do not hide these exclusions either. Report how many tasks were attempted, how many could be graded, which outcomes were excluded and why. A small clean subset should not be described as the whole benchmark, and a broken harness should not be silently repaired into a more flattering denominator.

Read the result at the right level
Observed outcomeWhat to investigateSafe conclusion
Test collection errorVerifier or environment setupThe intended behavior remains unverified
Relevant assertion failedPatch or hypothesisThe executed check rejected the attempted result
Relevant check passedCoverage and nearby behaviorThis check supports its stated criterion
No model participationHarness or service availabilityDo not score it as a model-generated solution

Completion must not depend on the final sentence

A model can confidently describe work that was not executed. It can also stop while a required approval, unfinished tool operation or failed check is still open. The runtime should use the recorded state to challenge a completion claim instead of treating a fluent final answer as the source of truth.

The current kernel includes completion and policy checks. The project also has tests for vacuous verification: a check that gives the appearance of success without meaningful evidence. These are useful implementation safeguards. Their presence does not remove the need to inspect the enabled policy, the chosen verifier or the scope of the task.

A blocked result is sometimes the correct result. If approval is missing, the environment is unavailable or the check cannot be run, the agent should explain what remains and stop at that boundary. Reporting ‘not verified’ is more useful than turning an obstacle into an invented pass.

Make the handoff useful to the next person

A good handoff identifies the changed files, the reason for the change, the checks actually run and the remaining limits. Include the command and relevant output, not just the word ‘tested.’ If a test was skipped, say why. If only one environment was checked, name it. Someone reviewing the work should be able to see where the evidence ends.

For a model comparison, add the checkpoint, runtime configuration, dataset version and sample count. Changes in serving mode or output-token limits can change measured accuracy without demonstrating a training improvement. Aditya’s recorded language evaluations are published with these caveats and language slices rather than as a single universal score.

Verification is not a promise that software will never fail. It is a disciplined connection between a claim and the evidence collected for it. The most useful coding agent is one whose work can be checked and challenged—and whose uncertainty remains visible when the check is incomplete.

Use several checks without making them mean the same thing

A compiler, a unit test, a browser interaction and a screenshot answer different questions. Typechecking can reject invalid interfaces and some inconsistent assumptions. A unit test can exercise a particular behavior. A browser check can establish that an interaction works in a tested environment. A screenshot can reveal clipping or poor hierarchy at a chosen viewport. None should be described as a substitute for all the others.

For a responsive interface change, begin with the changed behavior: a menu should open, expose its links, close with Escape and restore focus to its trigger. Exercise those interactions, then inspect the layout at relevant widths. A successful build does not establish focus behavior. A screenshot of an open menu does not establish that keyboard users can reach it. Keep each observation attached to the claim it actually supports.

The same principle applies to backend work. A passing isolated test may not exercise a real database or an external service. A local integration check may depend on fixtures unlike the production workload. Name those conditions rather than hiding them behind ‘all tests passed.’ If the environment cannot be reproduced, make that a limit of the handoff, not an excuse to invent a broader result.

Choose the checks in proportion to risk. A small wording change does not require the same release evidence as a migration that can remove data. Conversely, a high-risk action does not become low risk because several unrelated checks are green. The useful question is whether the selected evidence can reject the particular failure the operator needs to avoid.

Make the evidence reproducible without exposing private material

A reproducible handoff does not mean publishing everything collected during a run. Raw logs can include source code, personal data, service addresses or credentials. Keep the minimum evidence needed to inspect the claim, and review it before sharing. For a model evaluation, an aggregate summary can include sample counts, scoring rules and source digests while leaving raw prompts and generations outside the public artifact.

Digests help identify an exact source file, but they do not explain the experiment by themselves. Someone also needs the dataset identity, model configuration, method and exclusions. A matching hash can establish file identity; it cannot establish that the dataset was independent of training or that a verifier measured the intended behavior. Do not let a technically precise identifier stand in for a missing methodology.

Separate repeatable validation from a fresh experiment. Recounting saved correct answers can check the published arithmetic without running inference again. Rebuilding an interface can check that the release sources compile without exercising a live model endpoint. Both are valuable, provided the report says what was repeated. Calling a saved-result recount a new model evaluation would overstate the evidence.

Our Stage R v8 report follows this distinction: it publishes inspected historical counts, language slices and limitations, not a new inference run. The same standard should apply to coding-agent work. Record the revision, preserve the checks that actually ran, identify what the next reviewer needs, and leave uncertainty visible when a result cannot yet be independently reproduced.

Further reading

Aditya CodeApproval and safety boundariesStage R v8: recorded Indic evaluations