Every vendor of synthetic respondents reports an accuracy number. Most of those numbers are not comparable, because accuracy only means something against a baseline. The good news is that the research community has converged on a few ways to measure it, and a buyer can run most of them on a single past study.
Start with the ceiling: people disagree with themselves
Ask the same person the same question two weeks apart and you will not always get the same answer. That test-retest consistency is the ceiling for any simulation: no model can predict a person better than the person predicts themselves.
The Stanford study that built agents of 1,052 real Americans from two-hour interviews reported results this way. On the General Social Survey, the agents matched people's answers 68.85% of the time, while people matched their own earlier answers 81.25% of the time. Divided through, the agents reached 85% of the human ceiling. Agents built only from demographics reached 71%.
| Measure | Agents from interviews | Ceiling (people vs themselves) | Agents from demographics |
|---|---|---|---|
| General Social Survey answers | 0.85 of ceiling | 81.25% raw agreement | 0.71 of ceiling |
| Big Five personality | 0.80 of ceiling | r = 0.95 | 0.55 of ceiling |
| Economic games | 0.66 of ceiling | r = 0.99 | not better than the others |
Two lessons follow. Report accuracy as a share of the human ceiling, not as a raw percentage. And what the twins are built from matters: interviews beat demographic profiles by a wide margin.
100% = people matching their own answers two weeks later
Source: Park et al. (2024), Generative Agent Simulations of 1,000 People, arXiv v1
Compare distributions, not averages
A synthetic panel can get the average right and the spread badly wrong. OpinionQA, from Stanford in 2023, scored models on 1,498 Pew survey questions by comparing the whole distribution of answers with a distance measure suited to ordered scales. SimBench, a 2025 benchmark across 20 datasets, scores models from 0, no better than random, to 100, a perfect match. The best model it tested scored 40.8, and it found that the instruction tuning that makes assistants helpful improves consensus questions but hurts questions where people genuinely disagree.
Source: Hu et al. (2025), SimBench, arXiv 2510.17516
Seven checks to run
- Hold out a past wave. Give the vendor an old study with the answers hidden, have them simulate it, and compare.
- Score against the ceiling. Divide agreement by human test-retest, or by split-half consistency if you have no retest.
- Compare whole distributions per question, including the spread, not just the mean.
- Check subgroups. Find the best and worst served segments and look at the gap.
- Check relationships. Rerun your key driver analysis on the synthetic data and count coefficients that change sign.
- Test robustness. Shuffle answer options, reword the question, and rerun a month later.
- Check the ranking. Do the twins put your concepts in the same order the real people did?
Where the evidence is strongest
Ranking and direction hold up best. A large study of 70 pre-registered experiments, now published in Nature, found GPT-4's predicted effects correlated with real effects about as well as expert human forecasters, including for unpublished studies the model could not have seen, though it systematically overestimated effect sizes. A 2025 study with Colgate-Palmolive on 57 product surveys reached about 90% of human test-retest reliability on purchase intent, using a method that maps free-text answers onto the scale rather than asking for a number.
Questions to put to any vendor
ESOMAR's 20 questions for buyers of AI-based services are a good script. The three that matter most here: how do you verify and validate output for accuracy, and is it documented? What are the limitations of your models? How do you make it clear when AI is being used? A vendor who answers with a single accuracy number and no baseline has not answered.
Sources
- Park et al. (2024), Generative Agent Simulations of 1,000 People (arXiv v1)
- Santurkar et al. (2023), Whose Opinions Do Language Models Reflect?, ICML
- Hu et al. (2025), SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors
- Ashokkumar, Hewitt et al. (2026), Large language models can predict the results of social science experiments, Nature
- Maier et al. (2025), LLMs reproduce human purchase intent via semantic similarity elicitation
- ESOMAR (2024), 20 Questions to Help Buyers of AI-Based Services