Jean is now available in Claude. See how
Jean Technologies Icon
Jean Technologies Text

AI research

Consumer research costs billions a year. PyMC Labs and Colgate tested an AI panel against it.

On 57 real product surveys, a synthetic panel ranked product ideas about 90% as well as a second human panel would, once the model was allowed to answer in words instead of numbers. How it works, what it found, and where it stops.

Jonathan Politzki5 min read

Companies spend billions of dollars a year asking people what they think of products that do not exist yet. The most common version shows a few hundred people a product idea and asks one question: how likely are you to buy this? Every one of those panels has to be recruited and paid, each takes time, and a company can only afford to test so many ideas this way. The panels have their own problems too. The people who answer surveys are not everyone, and they tend to be kind to whatever they are shown.

A paper from researchers at PyMC Labs and Colgate-Palmolive starts from that cost and asks whether a language model can stand in for the panel. Their answer: not if you ask it the obvious way, and very nearly if you change how you ask.

The problem: asked for a number, the model says 3

A concept test shows people a product idea and asks how likely they are to buy it, on a scale from 1 to 5. Consumer goods companies run these by the thousand. The obvious shortcut is to give a language model a persona, show it the same concept, and ask for the same number.

That fails in a particular way. The authors report that the models typically answered 3, the safe middle, and almost never 1 or 5. The real panels answered mostly 4 and 5. So the synthetic answers pile up in the middle of the scale while the human ones sit near the top, and the two look nothing alike.

The method: let it talk, then score the words

The fix is to stop asking for a number. The authors call the method semantic similarity rating, or SSR. It has four steps.

  1. 1PersonaThe model is told who it is: an age, a gender, an income, a place.
  2. 2ConceptIt sees the same product slide the human panel saw.
  3. 3AnswerIt says in a sentence or two how likely it is to buy, with no number.
  4. 4ScoreThe sentence is compared with five reference statements, one for each point on the scale.

The reference statements run from "It's very unlikely that I'd buy it" to "It's very likely that I'd buy it". An embedding model measures how close in meaning the answer is to each one, and the closer it is, the more weight that scale point gets. So one answer is not one number. A hesitant sentence lands partly on 3 and partly on 4, the way a hesitant person might go either way.

Nothing is trained. The models were GPT-4o and Gemini 2.0 Flash as sold, and the survey data was used only to check the result.

What they tested it on

The data was 57 surveys of personal care product concepts, supplied by a leading company in that market, with 9,300 US respondents in total and 150 to 400 per survey. The paper asks two things of the synthetic panel.

First: do the concepts people liked more also score higher with the synthetic panel? Real panels are noisy, so the fair comparison is another real panel. The authors split each human panel in half at random, measured how well one half's scores track the other's, and report the synthetic result as a share of that ceiling.

Ordering the 57 concepts: share of the human ceiling reached (GPT-4o)
  1. Asked for a number directly81.7%
  2. Free answer, then the model picks its own number84.7%
  3. Free answer, scored by meaning (SSR)90.2%

100% = one half of a human panel against the other half

Source: Maier et al. (2025), Table 1, image stimulus with demographics

Second: does the spread of answers across the five points look like the human spread? This is scored from 0 to 1, where 1 means the two are identical. Here the method matters far more.

How closely the spread of answers matches the human panel (GPT-4o)
  1. Asked for a number directly0.26
  2. Free answer, then the model picks its own number0.72
  3. Free answer, scored by meaning (SSR)0.88

Source: Maier et al. (2025), Table 1. Kolmogorov-Smirnov similarity, averaged over 57 surveys

Two details are worth knowing. Gemini with SSR reached 90.6% on ordering and 0.80 on spread, so the result is not one model's quirk. And a conventional machine learning model trained on half of the surveys, using demographics and a few product features, reached about 65% on ordering. A general language model that had never seen the data did better than a model fitted to it.

Who the model is told to be matters

The 57 concepts scored much alike with real people, around 4.0 on average with little between them. That makes ordering them hard, and it is where the personas earn their place. When the authors took the personas away, Gemini still matched the human spread of answers, at 0.91, but its ordering of the concepts fell to 50% of the ceiling at best. A synthetic panel can look like a human panel without knowing which product people prefer. It needed to be told who was answering.

Not every part of the persona worked. The synthetic panel reproduced how purchase intent moves with age and with income. It did not reliably reproduce differences by gender or region, and the authors say personas cannot yet be treated as a reliable proxy for all subpopulations.

The answers come with reasons

Because the model answers in words, every rating has an explanation attached. One synthetic shopper said of a product, "It seems a bit too high-end for my needs and budget". The authors note that the real open-ended answers were often as short as "It's good". The synthetic panel was also less uniformly positive than the human one, which spread the concepts further apart.

Where it stops

  • It predicts what people say, not what they buy. The benchmark is a survey answer, and the paper has no sales figures.
  • It is about concepts, not people. The 90% describes each concept's average score across a panel. Nothing here predicts one person's answer.
  • One category, one country. The authors say the method is bounded by what the model learned in training, and personal care products are well covered there.
  • The reference statements were tuned by hand on these same 57 surveys. How well they carry to other surveys is open.
  • It is a preprint, with the code public.

Where it fits

The paper's case is about cost and scale. A concept test that needs a recruited panel and weeks of fieldwork can be run in minutes, on as many ideas as a team can write down, and still come back in the form a research team already uses: a rating from 1 to 5 for each concept, with the reasons attached. The authors say plainly that this is not a wholesale replacement for human research. The natural first use is the early screen, where twenty ideas need to become five before any money is spent on fieldwork.

The idea underneath travels further than purchase intent. A model is a better respondent when it answers the way a person does, in words, and the measuring happens afterwards. The paper's respondents are a general model handed a few facts about a person. What happens when the respondent is built from what real people said and did is the question we work on at Jean.