Evaluate a completed voice-agent conversation by checking the task outcome, the accuracy of the agent’s claims, and the evidence supporting each judgment. Keep transcript findings separate from questions that require audio, timing information, or a confirmed tool result. An overall score should remain traceable to the conversation and the rubric used to review it.
This guide proposes a provider-independent review workflow. It does not report a completed evaluation study or measured agreement between judges. The example rubric is a starting template, with no prefilled grades.
Start with the decision the review must support
Decide whether you are investigating a complaint, comparing two agent versions, checking policy compliance, or deciding whether a workflow is ready for wider use. Those questions can share some criteria, but they need different samples and different evidence.
For an appointment-booking assistant, distinguish understanding the requested date from actually creating the appointment. A polite confirmation is not sufficient evidence that a calendar operation succeeded. The reviewer needs the tool result or another authoritative record of the booking.
Write down what the agent was allowed to do, the reference policy in effect, and the expected outcome. Without those inputs, a reviewer may grade a reasonable-sounding answer against an assumption the product never intended to support.
Assemble a case that can be inspected later
Use a stable internal case ID and keep the provider’s call identifier as a separate field. Retain the version of the agent, the relevant instructions, the rubric version, and the origin of each evidence item.
| Input | What it helps establish | What it does not establish alone |
|---|---|---|
| Speaker-labeled transcript | What the available text says each participant said | Vocal tone, interruptions, or accurate acoustic timing |
| Audio, where available and appropriate | How the interaction sounded | Whether a downstream business action actually completed |
| Event timing | Delays between defined events | The cause of a delay without further traces |
| Tool requests and results | The operation attempted and the result returned | Whether the returned data was the correct policy reference |
| Policy and task context | What should have happened in this case | Whether the conversation followed it |
Provider event contracts differ. For example, Vapi documents an end-of-call report among its server events, while Retell describes configurable post-call extraction. Check the current event and extraction contract before treating a payload as a complete review case.
Only retain information you have a legitimate reason and appropriate permission to process. Redact unnecessary identifiers before creating an editorial example or sharing a case for review.
Give each criterion an observable definition
Avoid a single instruction to judge whether the call was “good.” Split the review into questions that a person can answer using the case evidence.
For the appointment workflow, a proposed rubric might include:
- Did the agent identify the requested service and date accurately?
- Did it distinguish a requested slot from a confirmed booking?
- Did its final claim agree with the available booking result?
- Did it explain the next step when the task could not be completed?
- Is there enough evidence to assess the criterion at all?
Allow “insufficient evidence” as an explicit outcome. Missing audio should not silently become a passing audio-quality grade. A missing tool response should lead to an investigation or an uncertain result, rather than a guessed success.
Separate transcript review from audio review
A transcript can support a judgment about an unsupported promise when the words are present and the supporting record is available. It cannot, by itself, establish how natural the voice sounded or how long a silence lasted.
Treat audio and timing as separate review tracks with their own prerequisites. Define exactly which timestamps a delay calculation uses. A measurement from a webhook delivery time is different from the time a participant experienced between speaking and hearing a reply.
For an illustrative missing-input case, a transcript may contain the words confirming a booking while the tool result is absent. The review should identify the unsupported evidence gap. That is a case-design example, not a result from an Agentium evaluation run.
Keep automated grades and independent review separate
A model judge can help triage cases against the rubric, but first inspect how its decisions compare with your intended judgments. Our comparison of model judges and human review explains the tradeoffs.
Give independent reviewers the displayed case without showing the automated verdict first. Record their rating, notes, and suggested improvement before resolving disagreements. Preserve both the original assessment and the final resolution so the history remains understandable.
Review a sample of apparent successes as well as flagged failures. A queue containing only model-flagged problems will not reveal every failure the model approved incorrectly.
Build a small, explicit ingestion workflow
For asynchronous review, separate receiving the completed event from running the evaluation. A proposed sequence is to validate the delivery, store a durable receipt, assemble the evidence, queue the assessment, and write the result with the rubric and agent versions attached.
Handle duplicate deliveries and incomplete evidence deliberately. Use the provider event identifier where the contract supplies one, or define a documented deduplication strategy. If enrichment fails, retain a visible incomplete state instead of writing a complete-looking score.
The same operating principles appear in our guide to reliable, idempotent webhooks, although billing events and conversation events have different data contracts.
Report findings the team can act on
Group problems by their likely owner: product instructions, reference material, tool behavior, audio configuration, or the evaluator itself. Keep the source evidence available when someone opens an individual finding.
Before a wider rollout, confirm that the team can explain an apparent failure, correct a mistaken grade, and reproduce which configuration produced the judgment. The useful outcome is an improvement decision supported by evidence, not merely a larger collection of scores.
Start with the example evaluation rubric, adapt its criteria to one workflow, and document which inputs each criterion requires.