Most Teams Don't Have a Data Flywheel

LangChain shows how the loop works. Here's why it stalls in production and what it actually takes to make it compound.

Austin Vance, CEO of Focused

LangChain has been pushing a clear idea: production data should make your agents better.

The loop looks like this: production traces capture real behavior, those traces become datasets, evaluators score performance, feedback improves those evaluators, and improvements get deployed back into the system. Over time, the system compounds.

That is the data flywheel.

And it is directionally right.

But most teams building agents today are not seeing that compounding effect. The loop exists on paper. In practice, it stalls.

What the Data Flywheel Actually Is

In the LangChain ecosystem, especially with LangSmith, the flywheel connects three things: observability, evaluation, and iteration.

Production traces become the source of truth. Failures are turned into datasets. Datasets become regression tests. Evaluators score performance at scale. Feedback improves those evaluators over time.

The goal is simple: every production interaction should become an improvement signal.

Where It Breaks

The issue is not the idea. The issue is that most teams never fully implement the system required to make it work.

1. Traces are collected, but nothing happens. Teams instrument their agents. They capture inputs, outputs, tool calls, and intermediate steps. And then it stops there. The missing step is turning traces into something actionable — structured datasets, labeled failures, repeatable test cases. Without that, you are not building a flywheel. You are just logging behavior.

2. There is no real evaluation layer. This is where most teams stall. They review outputs manually. They rely on intuition. They make changes based on what "looks better." There is no automated evaluation, no regression testing, no baseline performance. So when something changes, there is no way to know if it improved or regressed. If you cannot measure it, the loop does not spin.

3. Evaluators are not trusted. Even when teams introduce evaluation, it often breaks down. LLM-as-a-judge systems can scale evaluation, but only if they are clearly defined, calibrated against human feedback, and continuously refined. Without that, evaluator output becomes noisy. And noisy signals lead to random changes. If you do not trust your evaluation layer, you cannot rely on your flywheel.

4. The loop never actually closes. Even when failures are identified, prompts get tweaked ad hoc, changes are not versioned, and fixes are not tested against past failures. So nothing compounds. A real loop looks like this: a failure is captured, the failure becomes a dataset, the dataset is evaluated, a change is applied, and the change is tested against that dataset. If you skip any step, the loop breaks.

5. There is no real production pressure. This is the quiet failure that kills most flywheels. If your agent is not embedded in a real system, you do not get meaningful traffic, you do not see real edge cases, and you do not generate useful data. Internal demos do not create real signals. Without real usage, the flywheel has nothing to work with.

What a Real Data Flywheel Looks Like

At a system level, this is not a concept. It is a pipeline.

Instrumentation. Every step of the agent is observable — inputs, decisions, state transitions, outputs. Using structured systems like LangGraph makes this consistent.

Dataset creation. Production traces are turned into labeled examples, categorized failures, and reusable datasets. This is where the loop actually begins.

Evaluation. You define what "good" looks like and measure it — correctness, tool selection, completion quality. Evaluations run continuously, not just during development.

Calibration. Evaluators improve over time. Human feedback corrects them, agreement is measured, alignment increases. This step is critical and often skipped.

Iteration and deployment. Changes are applied intentionally — to prompts, graph structure, and tool logic. Then tested against historical failures before being deployed. Only validated improvements ship.

The Shift Most Teams Need to Make

The data flywheel is often described like a product feature. That is the problem.

It is not something you turn on. It is an engineering system that connects observability, evaluation, feedback, and deployment into a continuous loop. Without that system, you do not have a flywheel. You have logs and intuition.

The Bottom Line

Most teams do not have a data flywheel. They have a growing pile of traces and a sense that things might be improving.

The teams that actually get better over time treat this differently. They build the system that makes improvement inevitable.

If your agent only records what happened, it will stall. If your system learns from what happened, it compounds.

That is the difference.