Every company selling synthetic respondents publishes an accuracy number, and buyers naturally line them up and pick the biggest. That comparison does not work. Each number measures something different, against a different reference, at a different level of detail. As Semilattice puts it on its own comparison page, the figures are self-reported and use different metrics, so they cannot be compared directly.
What a buyer can do is understand what each number measures. That turns out to tell you more than the number itself.
The published figures
| Company | Published figure | What it measures | Level |
|---|---|---|---|
| Aaru | 3.53 points mean absolute error; 7.62% mean TVD | How far simulated answer shares are from real ones, across 2,993 questions | Population |
| Artificial Societies | 86% distribution accuracy | Overlap between simulated and real answer distributions, across 1,000 surveys | Population |
| Electric Twin | 92% | A distribution metric the company calls NDAM | Population |
| Semilattice | 0.775 ordinal alignment; 67.9% exact match | How closely each simulated person's answer matches theirs on ordered scales | Individual |
| Simile (Park et al. 2024) | 85% of the human ceiling | Agreement with each real person, divided by that person's own two-week consistency | Individual |
| Synthetic Users | 85 to 92% synthetic-organic parity | The company's own comparison of synthetic and real interviews | Interview |
| Listen Labs | Not published | Individual |
Three questions that decide what a number means
Is it a population or a person? Getting the share of yes answers right for a whole audience is much easier than getting each individual's answer right. Errors in opposite directions cancel out in a population. Aaru, Artificial Societies and Electric Twin report population numbers. Semilattice and the Stanford team behind Simile report individual numbers. A population figure of 90% and an individual figure of 70% can describe the same quality of model.
What is the baseline? A number means little without the floor and the ceiling. The floor is what a simple method gets: random guessing, or a general language model given a short biography. The ceiling is how consistent real people are with themselves when asked again. Artificial Societies is one of the few to publish both: a 67% floor for biography-prompted language models and a 91% ceiling for humans replicating themselves.
Who checked it? Every figure in the table is self-reported. Independent studies tend to find lower numbers, especially on new tasks. A Columbia mega-study of twins built from 500 answers per person found an average correlation of 0.20 with real responses across 19 new studies.
Where a number can be placed against the ceiling
When a source reports both its result and the human ceiling, the result can be put on one scale: the share of the ceiling it reaches. Only a few published figures allow this.
100% = real people matching their own answers when asked again
Source: Park et al. (2024); Maier et al. (2025); Artificial Societies evaluation page (86% and 67% against a 91% ceiling, divided through by us). Population and individual results are not the same task.
Even this chart mixes tasks: Artificial Societies measures populations, the Stanford study measures individuals, and PyMC Labs measures purchase intent. It is the closest thing to a fair comparison the published record allows, and it still needs a caveat on every bar.
What to ask a vendor
- Is this figure for a population or for individuals?
- What is the floor: how does a plain language model score on the same test?
- What is the ceiling: how consistent were the real people with themselves?
- Was the test data held out from everything the model was built from?
- Has anyone outside the company reproduced it?
We publish our own results, and the benchmarks we run other models on, on the open evaluations page. We will add Jean's figures to this table, with the floor and the ceiling, when we publish them.
Sources
- Semilattice, User simulation providers compared
- Aaru, Population simulation at the replication floor
- Artificial Societies, Method and evaluation
- Park et al. (2024), Generative Agent Simulations of 1,000 People (arXiv v1)
- Peng et al. (2025), A Mega-Study of Digital Twins
- Maier et al. (2025), purchase intent via semantic similarity elicitation
- Synthetic Users, science posts