Test voice and chat agents before customers do
Alyosha plays your customers, including the rushed, confused and angry ones. It calls or chats with your agent before launch and shows the turn where each conversation went wrong.
- Simulated calls
- 400
- Passed
- 297
- Failed
- 103
- Pass rate
- 74%
Call 0187, caller type: Rushed parent
- 1AgentThanks for calling Northside Clinic. How can I help?
- 2CallerHi, yeah, I need to move my son's appointment. The Thursday one.
- 3AgentSure. What is the patient's date of birth?
- 4CallerMarch 4th, 2017. Actually, can it be Friday instead? After three?
- 5AgentDone. I've cancelled Thursday's appointment. Anything else?Failed: cancelled without rebooking
- 6CallerWait, did you book Friday?
Rule: never cancel a visit without booking a new one or getting a clear OK. The agent lost the Friday request when the caller changed course mid-answer.
Pass rate by caller type
First run Rerun after the fix, same callers
Failed calls by rule
- Lost the request after an interruption61
- Read back the wrong date24
- No handoff when asked for a person18
Run it with
Methods
Scenario suite
Each call goal, such as rescheduling or a refund, is run by every caller type you get, so the test covers the mix you see in production.
Pass rate by caller type
Each conversation is graded pass or fail against your rules, and results are split by caller type instead of one average score.
Failed-turn review
Each failed transcript marks the turn that broke and the rule it broke, so your team fixes a cause instead of guessing.
Regression rerun
After a prompt, model or vendor change, the same callers run the same scenarios again, so you see what the fix repaired and what it broke.
Real-user check
Jean's AI interviewer has real people use your agent and then asks how it went, so you learn whether passing calls also felt helpful.
Callers who act like customers
Alyosha is trained on how real people talk and decide. Its callers interrupt, change their minds and give half answers, where scripted test cases and prompted LLM callers stay polite and on topic.
Pass rate by type of caller
Describe the callers you actually get. Results are split by caller type, so you see who your agent fails, not one average score.
The exact turn that failed
Each failed conversation points to the turn where it went wrong and the rule it broke. Rerun the same callers after a fix to see if it holds.
Questions
Does Jean work for voice and chat?
Yes. The same simulated caller can speak to a voice agent or type to a chatbot. You set the goal of each call and the type of caller.
Why not have an LLM play the customer?
A prompted LLM makes a cooperative customer. It answers every question in order and rarely changes its mind. Alyosha is trained on real people, so its callers bring the messy turns that break agents in production.
What counts as a pass?
You write the rules, such as verify identity before sharing details, never cancel without rebooking, or hand off when the caller asks for a person. Each conversation is graded against them, and every fail names the rule and the turn.
How do I regression test a voice agent?
Keep a fixed set of scenarios and caller types. Rerun it after every prompt, model or vendor change and compare pass rates by caller type. A fix for one caller often breaks another, and a split view shows it.
Do you monitor live calls or measure latency?
No. Jean supplies the simulated callers, the pass or fail on each conversation, and real-user checks before launch. It can sit next to the monitoring or voice platform you already use.
How do I start?
This product is early. Tell us what your agent does and who calls it, and we will set up the first test runs with you.