A user simulator for training agents
Agents that work with people need people in the training loop, and real ones are slow and scarce. Alyosha plays the user: it holds a goal, reacts turn by turn, and every episode can be scored and trained on.
Pass rate by user behavior
2,000 episodes, 400 per trait. Bars show pass^1, the share of single tries that pass. pass^4 is the share of tasks passed on all 4 tries.
- Clear and patient71%
- Low expertise52%
- Vague at the start41%
- Pushes back33%
- Changes mind mid-task9%
The agent fails most when the user changes the date after turn 3. Episode 0413 below is one of them.
Episode 0412
- Expertise
- low
- Vagueness
- high
- Pushback
- low
- Changes mind
- no
9 turns
Reward 1.0. Agent asked about seat and refund before booking.
Episode 0413
- Expertise
- high
- Vagueness
- low
- Pushback
- high
- Changes mind
- yes, date moves to Dec 13 at turn 4
12 turns
Reward 0.0. Agent booked Dec 12 after the change.
Episode 0414
- Expertise
- medium
- Vagueness
- medium
- Pushback
- medium
- Changes mind
- no
7 turns
Reward 0.5. Right flight, non-refundable fare.
Each episode exports as persona, hidden goal, transcript and reward.
Run it with
Methods
Persona grid
Simulated users spread across expertise, vagueness, pushback, patience and changes of mind, so your rollouts cover the users your agent will actually meet.
Hidden-goal tasks
Each task gives the simulated user a goal the agent cannot see, plus a check on the final outcome, so the reward is verifiable.
Episode export
Persona, hidden goal, full transcript and reward for every episode, as JSONL you can use for RL rollouts, fine-tuning or a held-out eval set.
Pass rate by user behavior
Success split by user trait, with pass^k over repeated runs, so you see which kinds of users break your agent and how often.
Check against real people
A slice of the same tasks run with real people through our AI interviewer, so you can see where the simulator and humans differ before you trust the reward.
A trained model of users
Most teams prompt a frontier model to play the user, and it plays a polite, patient one. Alyosha is trained on what real people say, do and why, so the agent learns against vague, impatient and changing users.
Hidden goals keep rewards honest
The simulated user knows the full task and the agent does not. The agent has to ask, and the outcome can be checked against the goal without leaking the answer.
Episodes you can train on
Each episode comes with its persona, hidden goal, transcript and outcome. Use them as rollouts for RL, as fine-tuning data, or as a held-out eval set.
Questions
Why not prompt a frontier model to play the user?
You can, and many teams do. Meta's AutoInteract used GEPA-optimized prompts on Kimi-K2.6 as its user. But Meta's MIMESIS work found that agents trained against a trained 4B/9B user simulator beat agents trained against GPT-5.5 as the user on 9 of 9 unseen simulators. Training the simulator on real people is the bet Alyosha makes.
Does the choice of simulated user change my agent's score?
Yes, often by a lot. Studies on tau-bench style benchmarks found that swapping only the simulated user moves task reward, and that simulators drift from their goal over long conversations. That is why we report pass rates by user behavior and check a slice of tasks against real people.
How do you stop the agent from gaming the simulator?
An agent trained against one narrow user learns that user's quirks. We vary personas across the grid, sample with real spread instead of one fixed style, and keep a held-out set of users the agent never trains on. The reward checks the outcome against the hidden goal, not how pleased the user sounded.
Does this work for UI agents or only chat?
Published user-simulator work for RL, including AutoInteract and MIMESIS, is conversational. Clicks and scrolls are less studied. Alyosha is trained on behavior as well as speech, but we will tell you plainly which settings we have tested for your task.
Can I buy the episodes as training data?
Yes. Real traces of people working with agents are hard to get. Jean can generate episodes for your tasks and personas, with transcripts, hidden goals and outcomes, for fine-tuning or evals.
Is this available now?
It is early. We are working with a small number of teams. Tell us your agent, your tasks and how you score them, and we will set up access with you.