User Simulation Leaderboard

31 simulators against 451 humans on 165 tau-bench tasks. Four behavioral components are Sorensen-Dice overlap with human-annotated behaviors, Eval is (1 - MAE) x 100 on outcome judgments, USI is the composite on 0 to 100. Numbers are mean over three annotation batches.

Mind the Sim2Real Gap, arXiv 2603.11245v2, Table 1 (mean of three annotation batches; 17 of 31 rows are in Table 1, the other 14 in Table 4, Appendix A.10). Reported by benchmark authors (third party to every system), 2026-03. Tool-transcribed, checked against the source table.

The same four USI behavioral dimensions, run by the provider on its own harness (two runs under manya-serve against three under verbatim for the comparison models, different episode coverage, run-level sample SD as error bars). Not the paper's harness, so it sits beside tau-USI and not inside it; the provider itself rules out an overall ranking from these numbers.

Persimmon launch post, 2026-09-10 (provider-run; means with run-level sample SD). Reported by the provider (Persimmon's makers), for every row, 2026-09-10. Tool-transcribed, checked against the source table.

Mean rate at which a judge is fooled into taking the simulator for the human; 50 percent is human parity. Provider-run, provider-defined.

Persimmon launch post, 2026-09-10 (provider-run; means with run-level sample SD). Reported by the provider, for every row, 2026-09-10. Tool-transcribed, checked against the source table.

Whether the simulator reveals a fact at the turn where the real human revealed it. Precision is the share of revealed facts that the human also revealed at that turn; recall the share of the human's reveals the simulator matched. Provider-run.

Persimmon launch post, 2026-09-10 (provider-run; means with run-level sample SD). Reported by the provider, for every row, 2026-09-10. Tool-transcribed, checked against the source table.

23 human-simulation tasks across five axes (CONV discourse, SS social skills, COG theory of mind, ROLE persona and role-play, EVAL judgment), each normalized to 0 to 100 and averaged unweighted. 14 of the 23 are judge-scored; a handful score real replies, survey answers or preference judgments. No human row. From Table 2 of the paper; the 'best specialized baseline' column is not a single system and is omitted.

OdysSim, arXiv 2606.14199v2, Table 2 (0 to 100, higher is better). Reported by benchmark authors, who are also OSim-8B's makers, 2026-06. Transcribed by hand.

20 human-behavior datasets unified; the SimBench score is the improvement of a model's predicted answer distribution over a uniform baseline, relative to the human distribution, by total variation distance: 100 is perfect alignment, 0 is random. Population level, stated answers. Table 1 of the paper.

SimBench, arXiv 2510.17516v4, Table 1. Reported by benchmark authors (third party to every system), 2025-10. Tool-transcribed, checked against the source table.

Six datasets of real recorded responses (news comments, book reviews, opinions, political text, chat, email) from about 23,000 users; response alignment score, higher is better. Table 1 of the HumanLM paper; the paper's own system is among the rows.

HumanLM, arXiv 2603.03303, Table 1 (response alignment, higher is better). Reported by benchmark authors, who are also HumanLM's makers, 2026-03. Tool-transcribed, checked against the source table.

200 users from a large short-video platform, three months of real logs, about 8,000 actions each across five scenarios and 22 action types. Binary behaviors by F1, continuous by normalized error, textual by a judge. Overall is the paper's composite. Table 1. Sits here for its framing as user simulation; its data is a platform behavior log, so it neighbours the RecSys track.

OmniBehavior, arXiv 2604.08362v1, Table 1. Reported by benchmark authors (third party to every system), 2026-04. Tool-transcribed, checked against the source table.

360,000 labeled actions over one month of continuous phone use by 20 users. Score is a judge model's similarity between predicted and true next action on 0 to 1; split is temporal within each user (weeks 1 and 2 train, 3 validate, 4 test). Table 3, mean over the 20 users. The paper's own system is among the rows.

LongNAP, arXiv 2603.05923, Table 3, mean over the 20 users (the last row). Reported by benchmark authors, who are also LongNAP's makers, 2026-03. Tool-transcribed, checked against the source table.

Recall@10 on ten Amazon categories, as reported in the OpenOneRec repository README (cross-domain transferability table). 'Ours' is the OneRec foundation model.

OpenOneRec repository README, RecIF-Bench results table (cross-domain transferability table, Recall@10). Reported by benchmark authors, who are also OneRec's makers, 2026-01. Transcribed by hand.

Eight tasks in a four-layer hierarchy over 100M interactions from 200k users across short video, ads and product. Recall@32 for recommendation tasks, AUC for label prediction, a judge score for item understanding and explanation. From the OpenOneRec README results table.

OpenOneRec repository README, RecIF-Bench results table. Reported by benchmark authors, who are also OneRec's makers, 2026-01. Transcribed by hand.