Synthetic respondents are useful and they fail in predictable ways. Knowing the failure modes is what lets you use them for the right jobs. Every point below comes from published research; sources are at the end.
Averages look right; spread and relationships do not
Political scientists at Vanderbilt compared synthetic answers with the American National Election Studies. The averages were close. The standard deviations were far smaller than in the real data, and when they reran standard regressions, 48% of coefficients differed significantly from the real ones, and among the significant ones 32% flipped sign. The same prompts also gave different results three months apart.
Source: Bisbee et al. (2024), Political Analysis
The practical lesson: a synthetic panel can tell you roughly where the middle is, and still mislead you badly on who differs from whom and why.
Groups get flattened
A study of 3,200 participants across 16 identity groups, published in Nature Machine Intelligence in 2025, found that language models misportray and flatten groups, giving a narrower, more stereotyped picture than the people themselves. Mitigations reduced the problem but did not remove it.
Twins of real people are better, but not solved
Grounding twins in a real person's data helps. The Stanford interview study reached 85% of human test-retest consistency on standard survey questions. But a Columbia mega-study of 19 pre-registered studies, with twins built from over 500 answers per person, found an average correlation of only 0.20 between twin and human responses on new, varied tasks, and named five distortions: too little individuality, stereotyping, representation bias, ideological bias and hyper-rationality.
These results are not in conflict so much as measuring different things: familiar survey instruments against a self-consistency ceiling, versus novel tasks scored by raw correlation. The honest reading is that twins are strong where the question resembles what the person was asked before, and weaker the further you move from it.
| Study | Twins built from | Tested on | Result |
|---|---|---|---|
| Park et al. 2024 (Stanford) | Two-hour interviews, 1,052 people | Standard survey and personality items | 85% of human test-retest consistency |
| Peng et al. 2025 (Columbia) | 500+ answers per person | 19 new pre-registered studies, 164 outcomes | Average correlation 0.20 with humans |
Too positive, too agreeable
Models trained to be helpful lean toward agreeable answers. Nielsen Norman Group found synthetic users claimed to finish every online course, where real learners admitted quitting after three of seven modules. A 2026 marketing study found synthetic answers were systematically more positive and less varied than real ones and significantly overestimated choice of well-known brands.
Sensitive to wording and order
A NeurIPS 2024 study of 43 models found a strong bias toward whichever answer is labelled A. When the order was randomized, answers drifted toward uniform randomness. Change how a question is worded or ordered, and a synthetic panel can change its mind in ways a real one would not.
Hidden assumptions in simulated experiments
When you vary one thing in a prompt, such as price, the model may silently assume other things changed too, such as competitor prices. Researchers at Columbia call this a confounding problem: the effect you measure is not the one you designed.
Less accurate further from the training data
Model answers resemble people in countries culturally close to the US much more than people in distant ones; one study found a correlation of minus 0.70 between cultural distance and similarity. SimBench likewise finds models struggle most with specific demographic groups. If your market is outside the populations the model or the twins were built from, expect lower accuracy.
Where they work
- Ranking and direction. Predicted effects of real experiments correlate strongly with actual effects, about as well as expert forecasters.
- Purchase intent screening. One study on 57 product surveys reached about 90% of human test-retest reliability.
- Demand curves and willingness to pay as a first pass, especially when tuned on prior human data from the same category.
- Early screening: narrowing ten concepts to the two worth real fieldwork.
How to use them safely
Use synthetic results to narrow, and real people to decide. Ground twins in real interviews rather than descriptions. Check spread and subgroups, not just averages. Disclose which findings rest on people and which on twins. Every study you run with real people is also the best test of the twins you ran next to it.
Sources
- Bisbee et al. (2024), Synthetic Replacements for Human Survey Data? The Perils of Large Language Models, Political Analysis
- Wang, Morgenstern, Dickerson (2025), Large language models that replace human participants can harmfully misportray and flatten identity groups, Nature Machine Intelligence
- Park et al. (2024), Generative Agent Simulations of 1,000 People (arXiv v1)
- Peng et al. (2025), A Mega-Study of Digital Twins Reveals Strengths, Weaknesses and Opportunities for Further Improvement
- Kaiser et al. (2026), Leaving Insight to Digital Twins?, NIM Marketing Intelligence Review
- Nielsen Norman Group (2024), Synthetic Users
- Dominguez-Olmedo, Hardt, Mendler-Dünner (2024), Questioning the Survey Responses of Large Language Models, NeurIPS
- Gui and Toubia (2023), The Challenge of Using LLMs to Simulate Human Behavior: A Causal Inference Perspective
- Atari et al. (2023), Which Humans?
- Hu et al. (2025), SimBench
- Ashokkumar, Hewitt et al. (2026), Nature
- Maier et al. (2025), purchase intent via semantic similarity
- Brand, Israeli, Ngwe, Using LLMs for Market Research, Harvard Business School working paper