Evaluating Voice Agents with LangSmith: Execution, Outcomes, and Experience

A voice agent finishing a call doesn't mean it worked.

‍

A scheduling agent can call the right tools in the right order and still book the wrong time, because nothing told it to confirm the caller's timezone. Another one can resolve the request and still make the caller sit through long pauses and repeated questions. Both calls failed. They failed differently.

LangChain's post on evaluating voice agents splits "did it work" into three questions, and that split is the useful part:

‍

• Execution. Did the agent follow its instructions, including the right tools, the right order, and the policies?
• Outcome. Did the interaction get the caller what they called for?
• Experience. Was the call smooth, responsive, and natural to sit through?

‍

I wanted to see what that looks like in practice, so I built a small refund agent and ran evals against it. Six scripted calls, eleven evaluators. Everything below is either from their post or from what I hit building it.

Execution: did it follow instructions

Execution is the closest thing here to normal software testing. Did the agent behave the way it was built to behave.

‍

For a voice agent that usually means tool invocation accuracy, policy compliance, whether it collected the required information before acting, and whether the answer was right given what it had. The final answer is only part of it. An agent can land on the right answer while calling tools it didn't need, skipping a confirmation, or reading data it had no business reading.

‍

Use code where you can. If the expected behavior is explicit and machine-verifiable, don't pay an LLM to check it. Tool call order, argument values, required disclosures, status codes, excessive tool usage: all of that is a code evaluator.

‍

My version of this is one function. The agent has to look the customer up before it files anything:

‍

def verified_before_refund(run, example) -> dict:
    tool_calls = _outputs(run).get("tool_calls", [])
    if "create_refund_request" not in tool_calls:
        return {"score": 1, "comment": "No refund filed, check not applicable."}
    lookup_idx = tool_calls.index("get_customer_info") if "get_customer_info" in tool_calls else -1
    refund_idx = tool_calls.index("create_refund_request")
    return {"score": 1 if 0 <= lookup_idx < refund_idx else 0, "comment": f"tool_calls={tool_calls}"}

‍

That's fast, free, and it gives the same answer every time. There's no reason to hand it to a judge.

‍

Use an LLM judge when meaning matters. Whether the agent explained the next step, recognized an ambiguous request, or followed a policy written in prose. Keep the task narrow and the rubric specific. "Was this a good response" is not an evaluator. "Pass if the agent confirms date, time, and timezone before calling the booking tool, fail if any field is missing or the booking happens first" is.

Outcome: did it get the caller what they wanted

An agent can execute perfectly and still fail the person on the phone.

‍

Back to the scheduling example. It collects a date and time, checks availability, books it. The workflow never mentions timezone, so the appointment lands in the wrong hour. The agent followed instructions. The instructions weren't enough.

‍

An outcome evaluator asks whether the request was resolved, whether the task was completed rather than described, whether the fallback was right when it couldn't be completed, and whether the agent had the context and tool access it needed.

‍

An LLM judge can read that off the conversation. When a business system knows the answer, use the business system instead.

‍

‍

The agent saying "you're all set" isn't proof. The ticket is.

Experience: was the call smooth

This is the part that transcript-only evaluation misses, and it's where most of my time went.

‍

Don't roll latency into one number. Trace each stage: time to first audio, speech-to-text, model time to first token, tool latency, text-to-speech. Measure P50, P95, P99. The median tells you what a normal call feels like. The tail is what makes an agent feel unreliable.

‍

One awkward pause could be transcription, the model, a slow tool, or audio generation. An aggregate number can't tell you which.

‍

I hit this immediately. My first responsiveness number was garbage because it included the ElevenLabs call, so a slow reply and slow audio looked identical. Timing them apart took two lines:

‍

started = time.monotonic()
result = agent.invoke({"messages": messages})
reply_secs = time.monotonic() - started      # model only

tts_started = time.monotonic()
speak(agent_reply, out_path=audio_path)
tts_secs = time.monotonic() - tts_started    # audio only

‍

Across six calls the model averaged 3.72s per reply. TTS added another 2.7 to 4.1s on top. Those are different problems with different fixes, and one number would have hidden that.

‍

If you're grading the voice, listen to the voice. A transcript tells you the wording was clear. It can't tell you the agent mispronounced a name, sped up, changed volume, or sounded robotic.

‍

This is the whole reason I went down this road. The transcript seems fine when reading it, but I know there's pauses in the response.

‍

So the audio judge gets the actual generated speech. Claude doesn't take audio input yet, so that one runs on OpenAI's gpt-audio:

‍

response = client.chat.completions.create(
    model="gpt-audio",
    modalities=["text"],
    messages=[{"role": "user", "content": [
        {"type": "text", "text": prompt},
        {"type": "input_audio", "input_audio": {"data": audio_b64, "format": "mp3"}},
    ]}],
)

‍

I split its output into two scores. Clarity covers pronunciation, volume consistency, and awkward pauses. Naturalness covers pacing, tone, expressiveness, robotic repetition, and whether the delivery fit the moment. Both come from one judge call per turn, cached, so the second metric is free.

‍

Look for conversational friction. Requests for repetition, clarification loops, interruptions and overlapping speech, long silences, abnormal call duration, early termination, failure to recover after being cut off.

‍

Context matters. A clarifying question isn't automatically friction. It's friction when the agent asks for something the caller already gave, or keeps asking the same thing.

‍

I made this a judge that returns a count plus the names of the signals that fired, so a score can be audited:

‍

class FrictionGrade(TypedDict):
    reasoning: Annotated[str, ..., "Which signals fired and where"]
    friction_events: Annotated[int, ..., "Total count of friction events"]
    signals: Annotated[list[str], ..., "caller_asked_agent_to_repeat, "
        "agent_asked_caller_to_repeat, repeated_clarification_loop, ..."]

‍

Some signals in their list I couldn't score at all: interruptions, overlapping speech, and long silences.

‍

All three need the same thing, which is both sides of the call on one clock. You need to know the caller started talking at 4.2s while the agent kept going until 4.9s, so those 700ms were overlap. Or that the agent stopped at 6.1s and the caller didn't start until 9.4s, so somebody sat through three seconds of dead air. Without both streams timestamped against a shared timeline, none of that exists as data to score.

‍

I've got two runtimes and only one of them can produce it. The live agent runs through Pipecat on a duplex transport, mic in and speakers out, with Silero VAD deciding when a turn ends. That side knows when both parties started and stopped talking. The eval harness doesn't use it. It replays scripted caller turns through the text agent and synthesizes each reply on its own, so there's no caller audio at all, just strings I typed. No second speaker means no timeline.

‍

So I left them unscored rather than invent a number, and the judge prompt tells it not to count them. Ask a judge to score interruptions off a transcript and it'll give you a number every time. The number is fiction.

‍

Closing that gap means running the evals through the Pipecat pipeline instead of around it. Synthesize the caller's turns as audio, play them into the transport on a schedule with deliberate barge-ins, and read overlap and silence off the VAD events both sides already emit. Turn tracking is on already for the tracing, so the events exist. I haven't wired the eval to drive them yet.

Match the evaluator to the claim

‍

Their post has a table for this and it's the right way to think about it:

‍

‍

A transcript can't prove pronunciation was clear. A judge is wasted on something a code check settles. A call that sounds successful doesn't prove the meeting got booked.

‍

What tripped me up

Two things I'd tell anyone starting this.

‍

The friction judge disagrees with itself. I ran the same six scenarios twice. The scripted repeat request got caught both times, since the caller literally says "Sorry, can you say that again?" The softer calls moved around: one scenario went from 2 events to 1, another from 0 to 1. If I keep going with this I'll run each scenario a few times and look at the spread.

‍

Report no data as no data. ElevenLabs ran out of credits partway through a run, so a batch of calls had no audio. My evaluator averaged an empty list and reported naturalness as 0, which reads as the worst voice you've ever heard. It was a billing problem. It returns None now with a comment saying why.

‍

What it looks like in LangSmith

‍

‍

Every turn breaks out stt, llm, and tts as its own span with its own timing, so when a reply felt slow I could see which one was slow. The feedback scores hang off the same run, which means a regression points at a specific metric.

‍

One caveat if you go looking right after a run: the per-run cells populate immediately but the average row lags about a minute. I spent a while convinced my evaluators hadn't attached before I checked the API and found all 66 feedback rows sitting there.

One score hides the tradeoffs

A prompt change can improve instruction adherence and lower resolution. A faster model can cut latency and get worse at recovering from interruptions. Roll it into a single quality number and you can't see any of that.

Track execution, outcome, and experience separately.