Meta trained coding agents against a simulated user. The user was a prompt.

At the end of October, researchers at Meta's FAIR lab published a method called AutoInteract, inside a longer report titled Aligning Autoresearch. The idea is simple. If you want an AI agent that works well with people, it should practice on people. Real people are slow and expensive to practice on, so the team built a simulated one.

This post covers what they did, what they left on the table, and what we found when we reran the first half ourselves.

What AutoInteract did

The team started from SWE-chat, a public set of real developers' sessions with coding agents. From those logs they wrote 189 user profiles: how expert each person is, how vague, how often they push back or change what they want. They widened those to 1,000 profiles. A simulated user then plays each profile through a coding task over several turns, and a small model is trained with reinforcement learning on the conversations that result.

It worked. On a multi-turn coding benchmark, a 4-billion-parameter Qwen model went from solving 23.0% of tasks on the first try to 43.2%. A 27-billion-parameter model went from 37.2% to 50.6%.

The user was a prompt

The simulated user is not a model trained to behave like a user. It is Kimi-K2.6, a general open-weight model from Moonshot, handed a profile and one carefully written instruction. That instruction was tuned with GEPA, an optimizer that rewrites a prompt again and again and keeps the versions that score better. The model's weights never change.

The paper does not say why it chose Kimi. Our read is that the choice matters. The authors note that models tuned to be assistants come out unusually cooperative when asked to play a user, and that a stronger assistant can make a worse simulator. A model that has been polished less heavily into a helpful assistant collapses less into one voice. It is more willing to be terse, vague or annoyed, which is how users are.

Even so, nobody trained a simulation model. The whole user side of the loop is a borrowed model and a prompt. That is the part we wanted to test.

What we found when we reran it

We reran the first half of the method: build the simulated user, then check it against real people. The paper prints no number for this step. It only shows the gain at the end.

Our version used WildChat, an open set of real people's conversations with a chatbot, in place of SWE-chat, and GPT-4.1 mini in place of Kimi. The test was plain. Given a real conversation so far, write what this person says next, for 120 people the simulator had never seen. Then compare with what they actually said.

First finding: the optimized prompt did not beat the plain prompt it started from, on any measure we tried.

Second finding: real people push back, and prompted simulators almost never do. In our test set, 16% of real follow-up messages corrected the assistant or told it that it had failed. The prompted simulator did that 1% of the time. Small open models that we fine-tuned on 1,500 real user turns from other people did it 9 to 11% of the time.

UserLM-8b is Microsoft's model trained to play the user side of a conversation. It was already close to real people before we touched it, which is the same lesson from another direction: a model trained on what users say beats a model told to act like one.

On a combined measure of how far simulated messages sit from real ones, the trained simulators landed between 0.10 and 0.12. The noise floor, the distance between two samples of real people, is 0.10. The prompted simulators sat at 0.20 to 0.23, about twice the floor.

Third finding: the judge is backwards. The usual way to score a simulator is to ask another model which message looks more real. We showed our judge a real message and the prompted simulator's message, blind. It picked the real one 4% of the time. Against the fine-tuned models it picked the real one about half the time, which is what you would expect if it could not tell them apart. A judge that prefers model-written text will rank simulators of people in the wrong order, and a prompt optimizer that climbs that judge's score is climbing toward sounding like a model.

The limits are real. This is one dataset, of chat rather than coding sessions, about 120 people, a different model from the paper's, and one training run. We have not yet run the second half, where an agent trains against the better simulator and has to come out better itself. That is the result that counts, and it is next.

The same idea is showing up everywhere

AutoInteract is one of several recent results that point the same way, and most of the others train the simulator.

Persimmon, from humans&, came out in September as a research preview. It starts from a base model rather than an assistant, NVIDIA's Nemotron 3 Ultra, and is trained on conversations between people. On the company's own multi-user Turing test it fools the judge 18.6 to 21.1% of the time, where 50% would mean it cannot be told from a person. The best model they compared against managed 1.2%. Those are their figures.

StudentSim, from Microsoft Research and the University of Illinois, trains simulators of individual students: first on many students' records pooled together, then on each student's own. The authors report that it beats GPT-5.4 on both of their measures in chess, English writing and mathematics. They also closed the loop. A chess tutor trained against the frozen simulator was preferred by expert raters over a tutor trained against a GPT-5.4 simulator. That is the step AutoInteract's prompted user leaves open.

Diachronic is a new entrant that describes its work as foundation models to predict people, and reports early results for a first model called Kairos 1. And the field's own workshop, UserSim at NeurIPS in Paris this December, asked in its call for papers for small, open simulators that are specialized models rather than prompted ones.

Where we are going

Two things follow for us. The first is the second half of the replication: an agent trained against a trained simulator, scored against the same agent trained against a prompted one.

The second is new. We are exploring synthetic traces alongside reinforcement learning. A trained simulator can sit in the loop as the user an agent practices on. It can also write realistic sessions and conversations to train on directly. Whether traces written by a trained simulator teach an agent more than traces written by a prompted one is an open question. We have not shown it, and we would rather run it than claim it.

AutoInteract showed that even a prompted stand-in for the user makes an agent much better. The next question is how much more a stand-in that behaves like people can do.