AI-moderated interviews at scale: how to run them well
AI-moderated interviews let a team hear from hundreds of people in a week. When to use them, what a good study looks like, and what to ask any vendor.
Jonathan Politzki6 min read
AI-moderated interviews let a research team talk to hundreds of people in a week, one at a time, with real follow-up questions, and get back themes with counts and exact quotes. They suit short, open questions asked of many people, not a two-hour depth interview or an ethnography.
The method is no longer an experiment. Anthropic reports that its AI interviewer ran 1,250 interviews with professionals, and that a later round reached 80,508 people in 159 countries and 70 languages. It also runs a system that sorts very large numbers of conversations into named, counted themes without anyone reading them one by one. Together they are a useful model for qualitative research at scale.
What is an AI-moderated interview?
An AI interviewer talks with one participant at a time. It works from a discussion guide and chooses follow-ups from what the person just said, as a good moderator would. Anthropic's first interviews lasted about 10 to 15 minutes each.
Drawn by Jean. The stages follow Anthropic's description of Anthropic Interviewer: planning, interviewing and analysis.
When is it the right tool?
- Use it when you need the reasons behind a number from more people than your team can interview: reactions to a concept, why customers switch or leave, how a segment describes a problem in its own words.
- Use it before a survey, to find the answer options you would not have thought to offer.
- Do not use it where tone of voice and body language matter. Anthropic notes its text-only interviews could miss those cues, and that knowing an AI was asking may have changed what people said.
- Treat what people say about their own behavior with care. Anthropic reports that 65% of its participants described AI as mainly augmenting their work, not automating it, while its records of actual conversations showed a much more even split.
What does a good study look like?
- A person approves the guide. Anthropic's interviewer drafts the questions from the research goals, and researchers review and edit the plan before any interview runs.
- Every theme has a count and a share. Anthropic reports, for example, that 69% of its general workforce sample mentioned the social stigma of using AI at work. A theme without a base is an anecdote.
- Quotes are exact. Each one is the participant's own words and leads back to a full transcript.
- Small groups stay hidden. Anthropic's theme system never shows a group with fewer than a minimum number of people or conversations, because a small enough group can describe one person. In a pilot reported in August 2026, outside researchers saw only these groups, and an outside team tried to identify people in the released data and, per the dataset card, could not.
How are themes counted?
Analyzing open-ended responses at this volume works in steps: a model describes each conversation, groups similar ones, names each group and nests groups under broader ones. Each theme's count should then be computed from those assignments, not written by a model as a sentence, so it adds up and can be checked.
Drawn by Jean, after the steps Anthropic describes for Clio, now called Anthropic Insights. The example themes are made up.
Labels are still judgments. Anthropic's dataset card says that in a check of the original system, roughly 3% of conversations were not clearly described by the group they were put in. Read a few transcripts in every large theme before you quote its number.
An example study
This example is made up. A snack brand wants to know why first-time buyers of a new flavour do not buy it again.
- The question: why did you not buy it a second time, and what would change that?
- The guide: six questions drafted by the AI and edited by the insights lead, with one rule for follow-ups: when someone gives a reason, ask about the last time it happened.
- Fieldwork: 300 first-time buyers, about 12 minutes each, over three days.
- What comes back: themes such as price per pack, could not find it in store, or tired of the taste, each with the number and share of people who raised it and a few exact quotes, split by age group. A region with only four respondents is not reported on its own.
- Next: the top reasons become answer options in a survey that sizes them.
What should you ask an AI interview vendor?
- Does a person approve the guide before fieldwork?
- Are counts computed from coded answers, or estimated by a model?
- Are quotes word for word, and can each one be traced to its transcript?
- Are synthetic and real answers kept apart and labeled in every count?
- What is the smallest group a finding is shown for?
- What did participants consent to, and can their words be published?
How we approach it
Jean runs AI-moderated interviews for research teams on those rules. A researcher approves the guide, counts are computed in code, every quote is exact and linked to its transcript, and synthetic respondents, which can give a first read before fieldwork, are labeled and never added to counts of real people. See an example in churn interviews.
Questions research teams ask
- Do people open up to an AI interviewer?
- In Anthropic's first study, 97.6% of participants rated their satisfaction 5 or higher out of 7, and 99.12% said they would recommend the format. They were recruited through crowdworker platforms, so treat this as a signal, not a promise for your audience.
- Can I use Anthropic Interviewer on my own customers?
- Anthropic presents it as its own research tool, not a service for other companies. The 1,250 transcripts from its first study are public, and worth reading to hear what these interviews sound like.
- How do I learn from customer conversations without a privacy problem?
- Keep the raw words where they are, show results only for groups above a minimum size, and have someone outside your team check that no person can be picked out. That is the approach Anthropic describes in its August 2026 pilot.
Sources
- Anthropic (4 December 2025), Introducing Anthropic Interviewer: what 1,250 professionals told us about working with AI
- Anthropic, What 81,000 people want from AI
- Anthropic (12 December 2024), Clio: privacy-preserving insights into real-world AI use
- Anthropic (26 August 2026), Enabling independent research on how people use Claude
- Anthropic Interviewer transcripts, dataset card on Hugging Face
- Anthropic Insights pilot partner cluster data, dataset card on Hugging Face
Also on the blog