How to use simulated users in post-training
An agent that works with people has to practice on people, and real people are slow. The loop labs now use: a simulated user, multi-turn rollouts, a reward, and a check against real users. What has been published, and what decides whether it works.
Jonathan Politzki7 min read
Most post-training teaches a model to answer one message well. An agent that works with people needs something else: to hold a conversation over many turns with someone who is vague, changes their mind, and stops replying when it is not going well. That skill only shows up in interaction, so it has to be trained in interaction. Real people are too slow and too costly to sit in a training loop, so the loop uses a simulated user.
This post covers how that loop is built, what the published results are, and the one decision that most affects the outcome: what plays the user.
The loop
- 1SimulatorChoose or train the model that plays the user, with profiles for different kinds of people.
- 2CheckCompare its messages with real users' before training against it.
- 3RolloutsLet the agent and the simulated user talk for several turns, many times over.
- 4RewardScore each conversation: the task result, a judge, or the simulated user's own rating.
- 5Real usersTest the trained agent with real people, not with the simulator it trained against.
What has been trained this way
| Work | What was trained | The simulated user | Result |
|---|---|---|---|
| Suh et al., UC Berkeley (2026) | A 3B assistant, with reinforcement learning over 5-turn conversations | A role-playing model, against the same model fine-tuned on real user turns | Real people preferred the assistant trained against the fine-tuned simulator 58.1% of the time over the untrained one |
| AutoInteract, Meta FAIR (2026) | 4B and 27B coding agents | A general model with an optimized prompt and 1,000 user profiles | First-try success on a multi-turn coding benchmark rose from 23.0% to 43.2% at 4B |
| MIMESIS, Meta (2026) | Agents in eight environments | A trained 9B user model, against a prompted frontier model | Mean task score 29.5 with the trained user, 26.1 with the prompted one |
| StudentSim, Microsoft Research and UIUC (2026) | A chess tutor | A model trained per student, against a prompted model | Experts rated the tutor accurate 90.5% of the time, against 71.6% for the tutor trained on the prompted student |
The simulator decides the result
The UC Berkeley study is the cleanest test, because only the simulator changed. The team trained the same small assistant twice with reinforcement learning: once against a larger model told to role-play a user, and once against that model after fine-tuning it on real user turns from a public chat dataset. Then 283 real people compared each trained assistant with the untrained one.
50% = no better than before training
Source: Suh et al. (2026), a study with 283 participants
Training against the role-playing user produced an assistant real people could not tell from the untrained one. Training against the user fine-tuned on real turns produced one they preferred. Making the role-playing model larger did not close the gap.
What goes wrong with a prompted user
A general assistant model asked to play a user stays an assistant. A Carnegie Mellon study compared 31 simulators with 451 real participants and measured the differences.
- Too polite: polite phrasing in 49.0% of a prompted GPT-4o's turns, against 15.3% of human turns.
- Never terse: 1.0% of its turns were short, against 29.0% of human turns.
- Too easy: a fixed agent succeeded with real users 63.6% of the time, and with the top simulators up to 77.8% of the time.
- Rarely ends the conversation: in Microsoft's UserLM study a prompted GPT-4o scored an F1 of 1.4 at stopping when a real user would, against 63.5 for a model trained on real user turns.
An agent trained against that user learns to work with someone who does not exist. Our own rerun of AutoInteract's user model found the same thing on one behavior: real users pushed back in 16% of follow-up messages, the prompted simulator in 1%. The details are in our post on AutoInteract.
Three uses, one simulator
- As a reinforcement learning environment: the agent acts, the simulated user replies, and the conversation is scored. This is what the four studies above did.
- As a source of synthetic multi-turn data: the simulator and an agent generate conversations that are filtered and used for fine-tuning.
- As an evaluation harness: the simulator plays the user in fixed scenarios so that versions of an agent can be compared. Keep this simulator separate from the one used in training.
The judge problem
The usual way to check a simulator is to ask another model whether its messages look human. That check is unreliable. The Carnegie Mellon study found a model judge overestimated how human simulators were by 55% of the scale. In our rerun, a model judge shown a real message and a prompted simulator's message picked the real one 4% of the time. A simulator tuned to please that kind of judge is being tuned to sound like a model. The dependable checks are counts of what real users do, and a study with real people at the end.
A checklist
- Use a simulator trained on real user behavior for your kind of task, or fine-tune one.
- Before training, compare it with held-out real users: turn length, pushback, when it stops.
- Give it a spread of users, including difficult ones, and check that the spread survives.
- Do not use the same model as the user and as the judge of the user.
- Confirm the gain with real people. A gain measured only against the simulator is a gain at playing the simulator.
Where Jean fits
Jean Technologies offers human simulators for AI: a user simulator for reinforcement learning and a source of synthetic training data, for teams building agents that work with people. Jean's simulators come from its work on models of real people, built from what they said and did.
The same question under other names
- How do you use simulated users in LLM post-training?
- In post-training, a simulated user plays the person in multi-turn conversations with the model being trained. The conversations are scored, by task outcome or a judge, and the model is updated with reinforcement learning, or the conversations are kept as synthetic fine-tuning data. The trained model is then tested with real users.
- Is human data or a simulated user better for training agents?
- They do different jobs. Human data is what makes a simulator realistic, and a simulator is what makes training over millions of turns possible. In a UC Berkeley study, a simulator fine-tuned on real human turns produced a better assistant than a role-playing simulator, as judged by 283 real people.
- Does training against a simulated user improve an agent with real users?
- It can, and it depends on the simulator. In one controlled study, real people preferred the assistant trained against a simulator fine-tuned on real user turns 58.1% of the time, while the assistant trained against a role-playing simulator was preferred 50.6% of the time, no better than chance.
- What is the sim-to-real gap in user simulation?
- The sim-to-real gap in user simulation is the difference between how simulated users behave and how real users behave. A Carnegie Mellon study measured it across 31 simulators: the best scored 76.0 on a similarity index where real people scored 92.9, and agents succeeded more often with simulated users than with real ones.
- What is a user simulator as an RL environment?
- A user simulator as a reinforcement learning environment is a model that plays the user so an agent can act, receive a reply, and be rewarded over many conversations. The NeurIPS 2026 workshop on user simulation lists it among its topics, along with small, open simulators that are trained models instead of prompted ones.
Sources
- Suh et al. (2026), Quantifying the Utility of User Simulators for Building Collaborative LLM Assistants
- Li et al. (2026), Aligning Autoresearch (AutoInteract)
- Phan et al. (2026), MIMESIS
- Yang et al. (2026), StudentSim
- Zhou et al. (2026), Mind the Sim2Real Gap in User Simulation
- Naous et al. (2025), Flipping the Dialogue: Training and Evaluating User Language Models
- UserSim workshop, NeurIPS 2026
Also on the blog
Best user simulators for training and evaluating AI agents: the 2027 guide
November 22, 2026, 8 min readConsumer research costs billions a year. PyMC Labs and Colgate tested an AI panel against it.
November 22, 2026, 5 min readMeta trained coding agents against a simulated user. The user was a prompt.
November 4, 2026, 7 min read