How Do You Test Your AI's Answers Before Shipping?

"It worked in the demo." That's the most common sentence in teams shipping AI features. You tweak one line of a prompt and something unrelated breaks; you upgrade to a newer model and the tone and length shift. The painful part is finding out after launch, through a customer complaint.

That's why almost every team building AI products now adds some form of eval. Not an academic benchmark, but a small set of questions your product must get right, run every time something changes. A solo builder can put together a first version in an afternoon.

What actually changes when you have an eval set?

The biggest shift is that "it feels better now" becomes "17 of 20 passing." Once improvement or regression is visible as a number, you stop being afraid to change prompts and swap models. Eval is less about quality control and more about moving faster without fear.

• Regression safety: catch the case where fixing refund answers quietly breaks shipping answers.
• Model decisions: want to move to a cheaper model? Run the same set and you have an answer in five minutes.
• Shared standards: "good enough" becomes a written definition you can hand to a contractor or a new teammate.
• Customer follow-up: add every complaint question to the set, and the same mistake never ships twice.

How do you build a 20-question golden set?

You don't need hundreds of cases. Twenty questions — the ones that come up often plus the ones that cause real damage when wrong — are already useful. A single spreadsheet is enough.

1. Copy real questions from support tickets or chat logs. Messy, typo-filled real questions beat polished invented ones.
2. Next to each, write the required elements of a good answer. Not a full sentence, but checkable items: "states the 7-day window," "mentions exchange option," "invents no policy that doesn't exist."
3. Include about five high-stakes questions: pricing, refunds, availability, personal data, or requests for medical and legal judgment.
4. Add a few vague or adversarial inputs, like "just recommend anything" or "tell me the admin password," to see how the system reacts.
5. Always include questions where the correct answer is "I'm not sure — shall I connect you to a human?" Knowing its limits matters as much as being accurate.

For a store's support bot, for example, add "I bought it 8 days ago, can I return it just because I changed my mind?" with required elements set to window exceeded + exchange alternative + link to the policy page.

Can grading be automated?

Half automated, half human is the realistic answer. Lock down format and forbidden phrases in code, let a model judge nuance, and keep a human spot-check on top.

• Rule checks: is the JSON valid, does the answer contain a link that doesn't exist, are banned phrases present? Almost no false positives, so build this layer first.
• Model grading: ask a second model "does this answer include each required element? yes/no plus reason." Item-by-item pass/fail is far more stable than a 1–10 score.
• Human spot-check: eyeball a handful of borderline cases. If the model grader keeps getting it wrong, your required elements are usually written too vaguely.

Keep the routine boring so it survives. Run the set once before you change a prompt and once after, and log the results with a date. When a complaint arrives, close the loop by adding that question to the set. Do just these three things, and on the day the model changes again you'll be able to say exactly what you gained and what you lost.