Agent Traces Rewrite the Harness

The basic function of traces is to make the agent harness better. Until then, they are simply expensive evidence. Stacks of model calls and tool calls and retries, of rejected actions and user fixes, of the dollar amount spent in tokens. Someone has to look at them.

We have seen companies spend millions to store traces, name it “observability”, and then the same incident reappear just with prettier pictures (e.g. trace with annotations). The agent still loops, the underlying tool schema is still wrong, the prompt still forces a fixed-size answer that model forgot 3 steps ago, the evaluation still is focused on end answer instead of the long windy path that produced that answer.

The issue is the missing improvement loop.

LangChain put the current version of this plainly: traces are becoming the currency of long-horizon agent improvement. That should make every engineering leader a little uncomfortable. A live agent is already generating the dataset. The question is whether the organization has a way to turn that dataset into labels, eval cases, harness patches, release gates, and new deployments.

That loop is the new agent harness work.

A trace is a work order

Agent observability used to be about tracing and debugging (as we typically think of it) for model-based agents. But this is table stakes for engineering and operations in general. Observability for agents is actually about understanding how an agent works on a day-to-day basis, and how it arrives at a particular decision or answer.

That is table stakes. LangSmith's observability model splits the world into projects, traces, runs, and threads. So, for example, tracing out a failure in a project can be useful for the engineering team working on that project. To debug a long-horizon agent, however, that trace must be opened up to show the run-level activity, i.e. the individual calls that together make up the trace. And then the threads of conversation through traces that were initiated as multi-turn conversations must also be openable.

A user correction is a label. A rejected tool action is a label. A repeated request is a label. A long reasoning path leading to a trivial step is a label. A tool call that succeeds but produces an answer that is vague is a label. Live traces are an expensive piece of evidence, so traces of live agent work, traces from which we could learn to improve long-horizon agents, become an expensive dataset. A cost signal from such a dataset would be a trace in which heavy spend arrived at a place with no improvement. Such traces are best studied outside of finance meetings with coding agents, because coding-agent spend already belongs in the trace. Such study can produce tool descriptions, information about middleware, information about routing, information about evaluators, and so on.

LangChain is moving the loop into harness engineering

The latest innovations in the field focus on improving the model (because apparently that’s been neglected). In the Nemotron 3 Ultra playbook, LangChain kept the model fixed and changed only the harness: the system prompts, tool descriptions, and even the middleware around individual model calls and tool calls. Generation settings are left at vendor defaults. Yet even with a fixed model, bigger gains are to be had from changing the harness.

That is the part enterprises own.

One more important point. The model vendor can fine-tune the model to perform better on a task. And that is great for the vendor. But the live system has a local contract, with local tool semantics, and local policy boundaries. The local approval points, local data access rules, local latency and cost budgets, all of these things get encoded in the agent harness, and that is what live evidence proves about.

On the one hand, the harness playbook starts with evals, i.e. the same things one would fine-tune in order to improve a model via more prompt editing. But on the other hand, since harness changes against real traffic do not have learning signals, these too must go through the harness eval process, which are nothing but traces of agent behavior running through the agent’s live harness. Better Harness describes live traces as a high-throughput source of eval material. The only additional step is to tag, holdout, human review, and sample representative traces as with any other source of eval material.

And this is a perfectly reasonable thing to do: go through traces of Agent executions, cluster failure cases together, select a representative subset of those traces to test the improved harness functionality (i.e. to run those traces through the harness with the updated logic), holdout traces (if there are enough of them) aside for verification of lack of regression, go through all of those with human review for correctness of interpretation of failure, add tags or other identifying markers to the subset the team tests with to determine whether particular failure cases were resolved for particular changes to the harness, and then, as before, watch the new traces as they are produced by the system after the improved harness has gone live.

Trace mining produces the missing evaluation material

Opening one trace is trivial. But to get value out of traces, a team needs to mine a large number of them.

See work around fine-tuning a perceived-error classifier from Fireworks and LangChain on trace-mining for superior traces-for-learning in a Qwen fine-tuned judge, using chat-langchain and Fleet data. Their results found that, for the task of trace judging, a trace-mining based solution was 10 to 100 times cheaper at scale while matching or beating frontier models on the trace-judging task.

That product produces artifacts a reviewer can inspect:

• labels that explain what went wrong
• issue clusters that group repeated failure modes
• representative examples for offline evals
• holdout sets that protect generalization
• monitors that reopen a problem when it returns to the foreground
• evidence for deciding whether a failure opened by an agent belongs to the harness or to the model

The LangSmith Engine also maps traces to a structured development workflow through its product loop. The docs say it works from live traces to surface recurring issues, diagnose root cause, propose fixes, deploy evaluators, create dataset examples, and reopen issues if they resurface. All of this is also analogous to agent failures opening tickets for a human reviewer to inspect and debug. As such, trace review from such agents also surface named work with owners, with evidence, that can also be used to setup regression checks on the harness as well as the model.

Later that month, Focused argued that AI agent evaluation ends too early when it is restricted to pre-release experiments. That AI agent accuracy is an observability problem, evidence from live systems is required to prove that the system worked, follows naturally from this.

The agent harness changes after the trace

This post looks at how reviews of traces can be “focused” by having them end with a name for the update surface that will fix things. The kind of note that says “Model hallucinated” is essentially useless as it essentially says “we gave up here” and does nothing to actually help improve things.

The reason for tool misuse is that the tool contract (the documentation) and/or the schema and/or the return format of the tool are not correct. So it might be good to have a closer look at the field that is wrongly included in the schema, and why the returned blob by the tool cannot be used as is in the dashboard.

Looping points to the area of middleware. A cap. A checklist. State-aware interventions after the second search for the same thing. Do not rely on model to stop looping and then treat that as governance in the system prompt in the 7th paragraph below.

Bad reasoning budget: This is the issue of how teams route work to models. An inexpensive model can do classifier work. A more expensive model can do multi-step repair work. A model that is reading the same file over and over again deserves a harness intervention before the high bill becomes a personality trait of the model.

Perceived error (human judgment) that there is an error: These are situations where users may correct the AI agent, reject output, or even rephrase the same request over and over again until they get the desired response. In these scenarios, the traces should be used as training material for the next round of evaluation by the agent.

Domain gaps in the trace or issues not reproducible require fine-tuning the model after the problem has been understood from the traces generated before. Fine-tuning before resolving the root cause of a failure in a live system creates fossilized mistakes, pleasant to trip over as a well placed concrete boot.

Day one can be boring. The trace can create a record that an engineer can trust. Which traces contributed? Which label fired? Which eval case was added? Which holdout protected the change? Which harness patch shipped? Which release gate passed? Which new traces prove the failure stayed fixed.

Those are receipts, not vibes.

AI agent lifecycle management starts here

Similar views on the role of live traces in AI agent lifecycle design come from Honeycomb in its write-up on AWS AgentCore. Canvas on AgentCore is a full fledged application that has session, deployment version, runtime state, online and offline evaluation, tool usage, and so on, all tracked for agent based applications. Honeycomb says to instrument the agent from day one.

But keeping them connected after release is a much harder problem, i.e. AI agent lifecycle management.

The trace-to-harness loop gives that lifecycle a spine.

Unless an organization has a real AI agent lifecycle management practice in place, all the work can just become folklore: “I remember we had to fix this bad trace once...”, “Okay, I added this one docstring...”, “Oh yeah, we replaced that model for that edge case...”. A week or two later nobody is able to say whether any of that actually fixed the particular failure being investigated, moved its expression to a slightly different case, or just masked it with a newer and cleaner-sounding incorrect answer.

The improved system creates a useful organizational structure around the agent, and therefore improvements to it, which is of greater value to the buyer than an agent that works in a workshop (as all AI do by now). This structure is a simple linear path from trace-evidence, via evals and so on through to harness-updates that in turn make their way through release-gates, until traces prove that changes worked or not. Conversation IDs make the trace coherent across the path, but coherence is only the start. The more valuable structure is one that shortens that path.

The winning team will have the shortest path from trace evidence to a harness release that safely changes what the agent does next time.