Build an evaluation set before shipping an AI feature

Build an AI evaluation set around decisions the feature must get right, including when it should decline to answer. A collection of impressive prompts is not enough. Each case needs an expected behavior, the evidence available at the time, and a rule for deciding whether the result is acceptable.

Do this before prompt tuning becomes the main activity. Otherwise, the team can spend weeks improving examples without agreeing on what qualifies for release.

Write cases as product contracts

A useful case record includes the user request, authorized context, expected outcome, prohibited behavior, language, and failure severity. Store the relevant source version so a later content change does not make an old answer appear wrong for the wrong reason.

For a retrieval feature, distinguish finding the right evidence from answering with it. Amazon Bedrock’s evaluation documentation describes evaluations for retrieval and generated responses, as well as human assessment. That distinction is useful even if the product uses a different platform.

An expected answer need not be a single exact sentence. Specify required facts, acceptable omissions, and unsupported claims that cause failure. This allows natural wording without making correctness a matter of taste.

Sample the difficult work deliberately

Gather sanitized examples from the actual workflow with permission. Add constructed cases for uncommon but consequential failures. Keep those origins visible: a synthetic challenge case does not establish how frequently the same situation occurs in production.

Include short questions with missing context, contradictory sources, unavailable information, and attempts to cross permission boundaries. For Costa Rican and US users, test Spanish and English separately. A translated prompt may preserve vocabulary while missing how users actually name a business process.

Maintain one set for development and another held back for release decisions. Repeatedly rewriting prompts against every available case turns the test into a memory exercise. When a production failure becomes a regression case, record why it was added.

A hypothetical support assistant evaluation

Suppose a team starts with 60 hypothetical cases: 30 routine questions, 15 ambiguous requests, 10 questions with no supporting source, and 5 permission challenges. These counts are a planning example, not a recommended sample size or evidence of statistical coverage.

The assistant answers 54 acceptably. A headline score of 90% sounds encouraging, but suppose one failure disclosed restricted content. The release should remain blocked under a gate that prohibits such disclosures, regardless of the average.

Case type	Example acceptance rule
Supported question	Required facts match cited evidence
Missing information	States the gap without inventing an answer
Ambiguous request	Asks for the missing distinction
Permission challenge	Restricted content never reaches generation

Repeat variable cases and compare versions on the same inputs. One successful run is weak evidence for a feature whose outputs can change between runs.

Assign a reviewer and a release gate

Use deterministic checks for structured fields and prohibited actions. Have a domain reviewer judge business correctness. If a model grades outputs, periodically compare its grades with human decisions; an automated evaluator can share the candidate’s blind spots.

Start by writing a small set of cases that would change your release decision, then assign an owner to every disputed expectation. Bring that artifact to custom software development planning. When someone proposes a prompt change, the team can check which cases it fixes and which it breaks.