In short
- Test with real, messy questions from your own users, not tidy examples.
- Have people who do the job judge the answers, and count harmful answers separately from wrong ones.
- Measure how answers vary across rewordings and repeat runs, then keep testing after launch.
Many assistants are chosen after a demo and a few hand-typed questions. That shows how a model behaves on tidy questions from people who already know what they want. Your users will be tired, rushed and imprecise. The point of testing before launch is to find out how the model behaves for them.
1. Build your test set from real traffic
Collect a few hundred real requests, such as support tickets, emails, call notes or search logs, with consent and with personal details removed. Keep what makes them hard: missing context, two questions in one, spelling mistakes, stress. Group them by task, and give each task enough examples to show a pattern. Around fifty per task is a reasonable start.
For each test case, write down what a good answer does: what it should establish, what it should do, and what it must not guess.
2. Decide what failure looks like
A wrong answer and a harmful answer are different things. A harmful answer would leave someone worse off if they acted on it: a wrong deadline, false reassurance, or a fluent answer to a different question from the one they asked. Score answers on a simple scale, such as helped, neutral or harmed, and track the harmed rate as its own number. Two models with the same accuracy can differ a lot on harm.
3. Use the right judges
The best judges are people who do the job the assistant is meant to help with: caseworkers, nurses, advisers, support staff. Show them answers without saying which model wrote them. For large test sets a model can do a first pass, but have a person check a sample of its verdicts, especially borderline ones, and check again whenever you change the judge model.
4. Measure variation, not just averages
- Rewording. Ask each question in several phrasings. If the answer changes, your users will see that too.
- Repeats. Run a sample more than once. A model that is right 80% of the time consistently can be safer than one that averages 84% and swings between runs.
- Refusals. Check that the model declines what it should, answers what it should, and doesn't quietly swap a hard question for an easier one.
5. Count the cost of a correct answer
Price per token doesn't tell you what a deployment costs. Divide what you spend by the number of questions answered correctly, and include retries and the time people spend on escalations. This number, cost per solved task, decides whether a deployment is affordable. Our guide to what a model really costs covers it in more detail.
6. Plan the fallback before launch
Decide what happens when the assistant is unsure or a question is out of scope: a person, a phone line, a form. Make that route easy to find and track how often it's used. If nobody ever uses it, treat that as a warning sign, not a success.
7. Keep testing after launch
Models change. Vendors update them, prompts get edited and traffic shifts. Keep your test set private, rerun it every month on the exact model version you serve, and compare the results. There is more on this in how AI models change after launch.
What this won't catch
A few hundred test cases won't reveal rare failures, drift across long conversations, or deliberate misuse. If a rare failure could seriously harm someone, treat this as a first screen rather than a sign-off, and plan a deeper review.