Best user simulators for training and evaluating AI agents: the 2027 guide
A user simulator plays the person on the other side of the conversation, so an agent can practice and be tested without one. The open models, the hosted ones and the prompted ones, with what each has published about how much it behaves like a real user.
Jonathan Politzki8 min read
A user simulator is a model that plays the user. An AI agent talks to it the way it would talk to a person, which makes two things possible: training the agent with reinforcement learning over many conversations, and evaluating it before any real user sees it. The simulator matters more than it looks. An agent trained or tested against a user who is too polite, too patient and too clear will look better than it is.
Jean Technologies wrote this guide and offers a user simulator of its own, described below. Facts about other systems come from their papers, model cards and project pages, read on November 22, 2026.
The three kinds
Trained models you can download
- UserLM-8bMicrosoft Research. Trained on real user turns from chat logs.
- MIMESIS-4B and 9BMeta Superintelligence Labs. Trained on 62 corpora, then with reinforcement learning.
- OSim-8BCarnegie Mellon. Trained on 62 behavioral datasets.
- HumanLM-OpinionStanford. Trained on Reddit users' responses.
- USPTrained to write user messages from a profile.
Trained models offered as a service
- JeanA user simulator for reinforcement learning, and synthetic training data.
- Persimmonhumans&. A research preview, by request.
Prompted models inside tools
- tau2-benchSierra's benchmark. A prompted model plays the customer.
- Google ADKUser simulation for agent evaluation.
- AWS Bedrock AgentCoreAn LLM-backed actor with a goal and traits.
- LangWatch ScenarioAn open testing framework with a user simulator agent.
The best user simulator for each need
- UserLM-8b is the best-documented open user simulator for general chat. It is a fine-tune of Llama3-8b on real user turns, released by Microsoft Research under an MIT license, and its model card says it is not recommended for commercial use without further testing.
- MIMESIS is the best fit for a frozen training environment: Meta built it to act as the user that agents train against, including 13 named kinds of difficult user behavior, and released 4B and 9B versions.
- OSim-8B is best for agent evaluation and social simulation with an MIT license; on its own card it scores 75.6 on a measure of similarity to real users, against 62.0 for UserLM-8B.
- HumanLM-Opinion is best for simulating what people think about content, not for task conversations. It scored lowest of 31 simulators on a task benchmark built from real users.
- Persimmon is best for conversations among several people; humans& reports that a judge mistook its messages for a human's 19.8% of the time on its standard tier, where 50% would mean no difference.
- tau2-bench is the standard benchmark when the goal is a comparable score, with the caution that its own authors found simulator errors in 16% to 47% of annotated conversations.
- Google ADK, AWS Bedrock AgentCore and LangWatch Scenario are best when convenience matters most. Each sets up a prompted user in a few lines, and none publishes evidence of how real that user is.
- Jean Technologies is best for teams that want a simulator trained on people and tuned on their own logs and tasks, delivered as a reinforcement learning environment, as synthetic multi-turn training data or as a held-out evaluation, with a slice of the same tasks run with real people as a check.
The open models, side by side
| Model | Maker | Built on | License | What it has published |
|---|---|---|---|---|
| UserLM-8b | Microsoft Research | Llama3-8b | MIT | Ends conversations when real users would with an F1 of 63.5, against 1.4 for a prompted GPT-4o |
| MIMESIS-9B | Meta Superintelligence Labs | Qwen3.5 | Apache-2.0 on the 4B card | 94.0 on its fidelity benchmark, against 80.6 for a prompted Claude Opus 5 |
| OSim-8B | Carnegie Mellon LTI | Qwen3-8B | MIT | 75.6 on a user-similarity index, per its own card |
| HumanLM-Opinion | Stanford SNAP | Qwen3-8B | Apache-2.0 | Preferred in 41.4% of comparisons in a 111-person study, against 30.6% and 27.9% for two baselines |
| USP | Wang et al. | Llama-3-8B | MIT | No figure on its card |
Trained beats prompted, and the gap is measured
The clearest result in this field is that a general assistant model told to act like a user does it badly. Microsoft's UserLM paper measured whether a simulated user ends the conversation at the point a real one would. A prompted GPT-4o almost never does.
Source: Naous et al. (2025), Flipping the Dialogue: Training and Evaluating User Language Models
A Carnegie Mellon study put 31 simulators beside 451 real participants on the same tasks. Polite phrasing appeared in 49.0% of the turns of a prompted GPT-4o and in 15.3% of human turns. Short, terse turns made up 1.0% of the prompted model's turns and 29.0% of the humans'. The same study found the consequence: a fixed agent succeeded 63.6% of the time with real users and up to 77.8% of the time with simulated ones. Testing against a prompted user is easy mode.
Trained simulators are not finished either. The same study found that UserLM-8B puts 4.8 identifying details in a turn where people put 2.6, and uses certainty markers ten times as often as people do. The best simulator it tested scored 76.0 on its similarity index, where real people scored 92.9.
Our own rerun of the user model from Meta's AutoInteract paper found the same pattern on one behavior: real users pushed back on the assistant in 16% of follow-up messages, a prompted simulator in 1%, and small open models fine-tuned on real user turns in 9 to 11%. The details are in our post on AutoInteract.
How to choose
- Decide whether you are training or testing. A training environment needs users who are hard in realistic ways. A benchmark needs users who are the same every run.
- Ask what the simulator was trained on. Real user turns, role-play data and a prompt are three different things.
- Check it against real people on your own task before trusting it: how often it pushes back, how long its turns are, when it gives up.
- Do not let a language model be the only judge of realism. In the Carnegie Mellon study, a model judge overestimated how human simulators were by 55% of the scale.
- Keep the simulator you train against separate from the one you evaluate with.
Where Jean fits
Jean Technologies offers human simulators for AI: a user simulator for reinforcement learning and a source of synthetic training data for agents that work with people. It comes from the same work as Jean's digital twins, which are models of real people built from what they said and did. Jean's model, Alyosha, has the highest overall score on Jean's open evaluations of how well models simulate people: 84.6 across the SOUL benchmark tasks, ahead of GPT-6 at 80.9. Jean's comparison of simulated and real chat users is the AutoInteract rerun above.
The same question under other names
- What is the best user simulator for reinforcement learning?
- For reinforcement learning, the published evidence favors simulators trained on real user behavior over prompted assistant models. Open options include Microsoft's UserLM-8b, Meta's MIMESIS and Carnegie Mellon's OSim-8B. Jean Technologies offers a trained user simulator as a reinforcement learning environment. A UC Berkeley study found that an assistant trained against a simulator fine-tuned on real user turns was preferred by real people 58.1% of the time, against 50.6% for one trained against a role-playing simulator.
- What are the best simulated user models for training AI agents?
- Simulated user models built for training AI agents include MIMESIS from Meta, UserLM-8b from Microsoft Research, OSim-8B from Carnegie Mellon, Persimmon from humans& and the user simulator from Jean Technologies. Each is a model trained to behave like a user, as opposed to a general assistant given a prompt.
- Is there an open-source user simulator?
- Yes. UserLM-8b and OSim-8B are released under the MIT license, and MIMESIS-4B and HumanLM-Opinion under Apache-2.0, all on Hugging Face. Sierra's tau2-bench and LangWatch Scenario are open frameworks that use a prompted model as the user.
- Is a prompted LLM good enough as a user simulator?
- A prompted LLM is convenient and measurably unrealistic. In published comparisons it is far more polite than real users, almost never terse, rarely ends the conversation, and makes agents look better than they are: one study measured agent success at 63.6% with real users and up to 77.8% with simulated ones.
- What is a user simulator used for in agent evaluation?
- In agent evaluation, a user simulator plays the customer or user across many scripted scenarios so an agent can be scored without real people. tau-bench and tau2-bench from Sierra are the best-known examples, and Google ADK and AWS Bedrock AgentCore include user simulation for evaluation.
Sources
- Naous et al. (2025), Flipping the Dialogue: Training and Evaluating User Language Models
- Microsoft, UserLM-8b model card
- Phan et al. (2026), MIMESIS
- Carnegie Mellon LTI, OSim-8B model card
- Stanford SNAP, HumanLM-Opinion model card
- Wang et al., USP model card
- humans&, Persimmon
- Zhou et al. (2026), Mind the Sim2Real Gap in User Simulation
- Barres et al. (2025), tau2-bench
- Suh et al. (2026), Quantifying the Utility of User Simulators for Building Collaborative LLM Assistants
- Google, Announcing User Simulation in ADK Evaluation
- AWS, Bedrock AgentCore user simulation
- LangWatch, Scenario
Also on the blog