Andy Garcia · freelance dev
FRENES
Back to the blog
AI & Research

120 AI Respondents Reached the Same Conclusion as 604 Humans. None of the Rare Answers Showed Up.

October 19, 2026 6 min read

On October 16, Nielsen Norman Group published a short study by Rachel Banawa whose title gives away the result: "Synthetic Users vs. Human Respondents: Same Conclusion, Less Variation."

The setup is simple. In a survey run in April 2026, NN/g asked 604 tech and design professionals to read a one-sentence description of some work ("An individual gathers insights about people's needs and experiences and turns those insights into recommendations for how a product or service should be planned, built, and improved") and to name the one field it describes. In February 2026, the team put the same question to 120 synthetic respondents generated on Synthetic Users, a commercial platform that simulates research participants. Then it compared the two datasets.

Both said the same thing: "UX" is still the dominant name for this work. What the synthetic panel did not reproduce is everything around that answer.

I build products with language models every day, and I know the temptation this study speaks to. For a product like BidForge, the tool I built to read French public tender files for small businesses, the cheap shortcut is obvious: ask a model to play twenty small-business owners and interview them. So I read the method before the conclusion.

What the synthetic panel got right

The main finding survived. 84% of synthetic respondents used UX-related terms, against about 70% of the humans. NN/g reports the 14-point gap as statistically significant, but it does not change the practical conclusion of the original study: no alternative label has displaced "UX".

Job role still shaped the vocabulary. Among synthetic customer-insights professionals, 45% used the word "customer"; none of the synthetic UX professionals did. 30% of synthetic market researchers said "market", again against 0% of the synthetic UX professionals. NN/g reports both differences as statistically significant.

If your question is "what would most people in this group say?", the panel answered it.

What it missed: the long tail

The humans used more than 50 distinct labels, including "consumer-behavior research", "digital-experience strategy", "journey management" and "human-factors engineering". Most appeared once or a few times. The synthetic respondents produced 12 labels in total: six UX-related, and six others (customer experience, customer insights, data science, market research, product management, service design). None of the unusual or specialized terms from the human data appeared, such as "digital management" or "business analysis". Nor did "human factors", "experience design" or "consumer research", which the human answers contain.

NN/g calls this mode collapse: synthetic data that reproduces the most common answer and drops the long tail of less common ones.

One detail makes it stronger. The platform simulates interviews, so synthetic respondents gave longer answers than humans filling in a survey. More words, more chances to name an alternative. They still named fewer.

A caution on that count, though. The number of distinct labels grows with sample size, and 120 is not 604. The article does not say how many labels a random 120 of the humans would have produced. So I would not lean on "12 against more than 50" alone. The stronger evidence is elsewhere.

Built to be more diverse, it came back more uniform

NN/g's human sample was recruited partly through its own channels, including its Practitioner Panel, and 67% of it was UX professionals. The July report on the original survey already warned that this may overstate how common "UX" is. The synthetic sample was set up with quotas across roles, by design: only 20% were UX professionals.

You would expect a sample with less than a third of the UX share to use UX vocabulary less. It used it more.

Product managers show it best. 93% of synthetic product managers used UX-related terms, against 44% of human product managers. In July, NN/g had reported that 38% of human product managers used the word "product", against 10% of UX practitioners. NN/g's own chart shows the same pull across the whole panel: 3.3% of synthetic respondents used "product", against 12.4% of the humans, even though the synthetic panel held far fewer UX professionals.

Put the two NN/g pages side by side and the effect is not a simple flattening, either. In July, 18% of human customer-insights professionals used "customer". In the synthetic panel, 45% did. That comparison is mine, not NN/g's, and neither article gives the subgroup sizes. But it suggests the panel can amplify the obvious cue, the word in the job title, for one group while shrinking a real difference for another. Without human data next to it, you cannot tell which way it went.

What the study does not show

It is one open-ended question, on one platform, with synthetic answers obtained in February 2026. The article does not name the model behind the platform (the vendor describes a feature that shuffles between several large language models), and products like this change fast. The human "ground truth" is itself a skewed sample. Two aspects of the platform were outside NN/g's control, and it flags them as possible confounders: the interview format, and the platform sometimes rewording the question ("people's needs, behaviors, and experiences" instead of "people's needs and experiences"). NN/g says it cannot determine exactly why the synthetic respondents leaned so hard toward UX terms; it points to earlier research finding that synthetic answers cluster around common or expected responses.

So this is not proof that synthetic respondents fail at everything, or that another tool or prompt would fail the same way. It is consistent with NN/g's August 2025 review of three studies, which concluded that synthetic users "may capture trends in human behavior but not the magnitude of the effects or the variability in the human data." NN/g also warns against the obvious fix: prompting a model for more diverse answers produces variety, not necessarily the variety real people have, at the frequencies they have it.

The vendor's own FAQ agrees

This part is not in the study. On its site, read on October 19, Synthetic Users calls itself "a discovery co-pilot" and states: "Real user research stays essential for validation and edge-case work." Edge cases are the long tail. On that point, the vendor and the study say the same thing.

Two other lines deserve a second read. The site says that across its independent comparison studies, "parity sits between 85% and 92% depending on audience type, measured across thematic overlap, depth, and qualitative alignment." In NN/g's study, the overlap was real: same dominant theme. What went missing was the tail, and a score built on thematic overlap can stay high while the tail is gone. The FAQ also says that 10 to 12 well-defined participants "typically reach saturation", with a score showing "when new participants stop adding novel themes." With a generator that converges on a dozen labels, an early plateau may say more about the generator than about your market. Plans start at $12,500 a year.

What I would do with it

I fact-check my own posts with a model, Jev, used as a judge. That is a narrow job: a verdict on a claim, against a source I can open. A synthetic respondent does the opposite job. It invents what a person would say, with nothing to check it against. I would set the bar accordingly.

Before an interview round, I would use a synthetic panel to draft the interview guide, list the obvious objections and catch confusing wording. That fits the use NN/g leaves room for: generating hypotheses and checking broad patterns.

I would not use it to size a niche, to compare two segments, or to find the early adopter whose odd phrasing turns out to be the product. Those are the tail and the subgroups, exactly what went missing here.

In a real interview round, I would count the distinct answers, not only the top one, and keep the answers that appear once in their own column. That is where the next question comes from.

And if a decision that matters rested on a synthetic panel, I would ask the same question to even a small human sample before trusting any difference between groups. NN/g puts it bluntly: research with synthetic users "is still not user research, because no actual users are being studied."

A synthetic panel can tell you what most people would say. The customer you are looking for is often the one who says something else.

Sources

A project like this one?

I design and deploy products like this. Let's talk.

Let's talk