Evaluate an AI feature by defining what successful behavior means for the customer's task, assembling representative cases, and measuring a candidate against a recorded baseline. Include failures that matter to the business, not just examples that make the feature look impressive. The evaluation should help you decide whether to release, revise, or limit the feature.
Keep application correctness and model quality visible as separate concerns. A well-written answer is still an unacceptable result if it uses another customer's data or triggers an unauthorized action.
Define success in operational terms
Imagine a hypothetical SaaS that classifies delivery exceptions and drafts the next internal action. Its users need to distinguish damaged goods, incorrect addresses, delayed handoffs, and reports that lack enough information.
A useful result assigns a supported category, cites the relevant details, and asks for missing information when necessary. A harmful failure might invent a delivery commitment or recommend closing an unresolved investigation.
Anthropic's evaluation guidance recommends specific, measurable criteria connected to the application's purpose. Translate that principle into a short rubric for your workflow: what must be correct, what may vary, and what would block a release.
Build a representative case set
Collect cases that reflect the inputs you expect to support. Use appropriately authorized data or carefully constructed synthetic examples. Record where each case came from and what behavior it is intended to test.
For an illustrative starting set of 60 cases, you might include 24 ordinary reports, 12 with missing information, 10 with conflicting details, eight unusual but supported situations, and six attempts to redirect the system away from its task. This invented distribution is a planning example, not a recommended universal sample size.
Include realistic noise: misspellings, long explanations, multiple issues in one report, and an attachment that does not support the accompanying claim. Add languages and formats only if your product intends to support them, then report their results separately.
Protect a holdout from routine tuning
Separate cases used for development from cases reserved for checking a proposed release. Use development cases to inspect failures and improve prompts or workflow code. Keep the holdout outside routine tuning so it can provide a less rehearsed comparison.
Maintain case identifiers and version history. When a real failure becomes a regression example, record the expected behavior and why it matters. Avoid silently rewriting expected results to match the newest candidate.
As the product evolves, refresh the dataset to reflect new workflows and observed inputs. A holdout is useful only when it remains relevant. The objective is a fair view of performance on the intended task, not preserving an old score indefinitely.
Combine different kinds of checks
Use deterministic checks where the requirement is unambiguous. Confirm valid output structure, allowed category values, correct customer references, source identifiers that exist, and the absence of an unauthorized state change.
Use a rubric for qualities that require interpretation, such as whether a suggested action follows the evidence. Human review can resolve ambiguous cases and reveal that the rubric itself needs clarification.
An LLM judge can help review more outputs, but calibrate it against human judgments on a meaningful subset. Inspect disagreements and give the judge the evidence needed to apply the rubric. Do not treat agreement between two models as independent confirmation of factual correctness.
Report separate measures for different failure types. One overall score can hide a small but serious class of mistakes.
Compare a candidate with the baseline
Run the baseline and candidate with recorded prompts, provider configurations, workflow versions, and relevant generation settings. Track latency, usage, and failure disposition alongside quality.
Suppose, in a hypothetical comparison, a new prompt improves ordinary category selection but mishandles reports with conflicting dates. The aggregate score might improve while the workflow becomes less trustworthy for a consequential scenario.
Inspect that category before release. The response may be to revise the prompt, add a deterministic requirement check, require human review for conflicting dates, or restrict the supported input. The decision should follow the product consequence rather than the appeal of a higher average.
Repeat selected high-consequence or unstable cases when variability could change the decision. Choose repetition based on the question you need to resolve and the cost of running it; there is no magic number that proves reliability for every application.
Test the complete product behavior
The release check should include permission failures, unavailable providers, cancelled requests, invalid model outputs, exceeded budgets, and duplicate submissions. Verify how each outcome appears to the customer and the operator.
For the delivery classifier, a failed model call should not leave the investigation marked resolved. A rejected output should not be displayed as an approved next action. A background retry should retain the original workspace context.
Agentium Start's capability page describes the foundation and labels its automated test suite as partial. Treat AI evaluations and your product-specific release checks as explicit work to add and maintain, with their own evidence.
Record the release decision
Create a concise release record containing the evaluated version, dataset version, acceptance criteria, meaningful results, known limitations, and the person responsible for the decision. Explain any scope restrictions clearly in the product.
Keep the previous configuration available where rollback is appropriate, and plan what happens to operations already running. After release, review appropriately collected feedback and failures, then feed relevant cases into the next evaluation cycle.
Use the Agentium Start documentation to understand the surrounding application when building on its source. Release the AI feature when the evidence supports its specific use, with a repeatable process for judging the next change.
Authority References & Sources: