Are You Managing AI Quality by Gut Feeling?

Most small teams now ship at least one AI feature — a support chatbot, an auto-reply, a document summarizer. But ask "is it working well?" and the answer is usually the same: "I tried a few prompts and it seemed fine." When the model gets updated or someone tweaks one line of the prompt, nobody can say what improved and what quietly broke. That is why the word eval keeps coming up in 2026. Models are a commodity now; the differentiator is whether you can measure quality against your own use case.

Why isn't "trying a few prompts" enough?

Manual spot checks only confirm the cases you already expect to pass, because people unconsciously pick questions their prompt handles well. On top of that, the same input can produce a different answer on each run, so one good result proves nothing.

Here is a scenario that happens constantly. A store adds "be warm and friendly" to its support bot prompt, and the tone improves — but the bot starts describing the refund policy vaguely instead of stating the 7-day window. The tone test passed while the policy answer collapsed. This kind of regression is invisible without a fixed list of test questions.

How do you build a 20-question eval set?

You don't need an academic benchmark. Twenty real customer questions plus one or two lines describing what a passing answer must contain is enough. A single spreadsheet works.

1. Pull from real logs — chat transcripts, Instagram DMs, phone notes from the last two to four weeks. Invented questions never match the real distribution.
2. Cover the categories — 10 common questions, 5 ambiguous or incomplete ones, and 5 that the AI should refuse or defer (improvised discounts, medical advice, trash talk about competitors).
3. Write pass conditions, not model answers — for example: "must state the 7-day refund window," "must not guess stock levels; offer to confirm instead," "under 200 characters."
4. Keep adding failures — any conversation where a customer complained or a human had to step in goes into the set the same day. Those are your most valuable rows.

For a neighborhood pilates studio, the list might include "How much for twice a week for three months?", "Can I cancel yesterday's booking?", and "I have a herniated disc — is this safe for me?" The pass condition for the last one is simple: no medical verdict, recommend a consultation instead.

Should a human grade the answers, or can AI do it?

Use both. Grade rule-based things with code, judgment calls with an LLM-as-judge, and have a person sample-check the results. Don't automate on day one — grading 20 items by hand twice will show you exactly what is safe to automate.

• Rule checks: banned words, length limits, whether the phone number and links are actually yours. Regex is enough.
• AI grading: ask "did this answer meet the pass condition?" with a yes/no plus a one-line reason. Binary judgments are far more stable than 1–5 scores.
• Human spot check: eyeball about five graded items per run to confirm the grader itself hasn't drifted.

Operating it is simple: run the same 20 questions before and after any prompt or model change, and look only at the items whose scores dropped. The moment you can say "18/20 to 19/20, but one refund question broke," your deploy decision is made by data instead of by meeting.

What can you do today?

You can build a first version in 30 minutes. The most common failure is postponing the start while trying to design a perfect eval set.

• Paste 20 recent customer questions into a spreadsheet.
• Add one line of pass condition per row.
• Run your current prompt, grade the outputs, and record the score with the date.
• Re-run the same sheet the next time you touch the prompt.

Shipping an AI feature is easy; maintaining one is not. An eval set is the cheapest asset a small team can build to cut that maintenance cost in half.