Synthetic users agree with real ones, until the details matter

Synthetic users can tell you the most common answer, but they cannot tell you the full range of what real people think. That is the short version of a new Nielsen Norman Group study, one of the clearest tests of synthetic users UX research so far: AI respondents reached the same headline conclusion as people, then lost most of the variety that makes research worth doing ^[1].

For a studio like ours, that finding matters. Clients increasingly ask whether an AI panel can replace interviews to save time and budget. Here is what the evidence says, where synthetic respondents are useful, and how we fit them into a research plan without fooling ourselves.

Key takeaways

• In NN/g’s comparison, 120 synthetic respondents and 604 human professionals reached the same main conclusion ^[1].
• Humans used more than 50 distinct labels in their answers. Synthetic respondents used about a dozen ^[1].
• Differences between subgroups were exaggerated or flattened in the synthetic data ^[1].
• NN/g’s advice: use synthetic respondents for hypotheses and broad patterns, and real people when the long tail matters ^[1].
• Earlier NN/g tests found synthetic users too shallow and too positive to replace real-user research ^[3].

What the NN/g study compared

NN/g’s Rachel Banawa revisited an earlier survey about what to call the work UX people do. In that survey, 604 tech and design professionals named the one field that best describes a person who gathers insights about people’s needs and turns them into product recommendations ^[1]. In February 2026, she generated 120 synthetic respondents on the Synthetic Users platform and asked them the same open-ended question ^[1].

The platform assigned each synthetic respondent a profile with age, location, profession, cultural background and personality traits ^[1]. Both datasets were then coded the same way, by keyword, so the answers could be compared directly ^[1].

Where synthetic users UX research matched real people

On the headline question, the two groups agreed. UX terminology dominated in both: 84% of synthetic answers used it, against 70% of human answers ^[1]. The 14-point gap was statistically significant, but it did not change the conclusion ^[1].

“Both datasets led to the same primary conclusion: no alternative term has displaced UX as the dominant name for this type of work.”

That is the case for synthetic respondents in one line. If all you need is the most common answer, they can get you there quickly. NN/g says as much: when only the most common response matters, synthetic data may be enough ^[1].

Where it broke down: the long tail

The trouble starts below the headline. Human respondents used more than 50 distinct labels, including “journey management”, “digital-experience strategy” and “human-factors engineering” ^[1]. The synthetic answers collapsed onto about 12, and never mentioned common terms like “human factors”, “experience design” or “consumer research” ^[1].

Subgroups suffered too. In the synthetic data, 93% of product managers used UX terminology. Among real product managers, it was 44% ^[1]. NN/g links this to mode collapse, where a generative model produces a narrow band of likely outputs instead of the full spread a population would give ^[1]. The model did not erase the effect of job role, but it pulled every role toward the dominant answer ^[1].

The study has limits, and NN/g is open about them. The human sample leaned toward UX professionals, at 67% versus 20% in the synthetic sample, and the platform sometimes reworded the question ^[1]. Even so, the synthetic panel leaned harder toward the obvious answer than a sample that was already biased toward it ^[1].

This is not a new warning

In 2024, NN/g’s Maria Rosala and Kate Moran tested synthetic users against three real studies ^[3]. Synthetic users praised a drone-delivery concept as a game changer, while real users gave more critical feedback ^[3]. In a study on online learning, real participants admitted they had finished only some of their courses, while synthetic users claimed to finish all of them ^[3].

“Do not use synthetic-user research as a replacement for real-user research.”

Both studies point the same way. Synthetic respondents sound plausible, agree with conventional wisdom and smooth over the awkward cases. That makes them convincing in a slide deck and weak as evidence.

Why the long tail is where products are won

Most of the insight we act on in client projects comes from edge cases. The travel agent who books for six families at once. The guest who pays half in cash at the counter. The volunteer scanning tickets on a cracked phone in bad light. Those are exactly the people a synthetic panel averages away, and they are often the ones who decide whether a product works in the real world.

There is a wider pressure behind this. In an August article, NN/g’s Anna Kaley and Raluca Budiu describe a “custodial era” of UX, in which AI produces features and prototypes faster than teams can evaluate them ^[2]. When production outpaces evaluation, the gap becomes UX debt: things that demo well but confuse people in real use ^[2].

“Shipping something quickly is not the same as creating something useful.”

Their fixes are practical. Keep a user panel for quick testing, use evaluation templates and heuristic checklists, and give AI tools a UX-context file plus your design system to work from ^[2]. Replacing users with simulations works against all of that.

How your team should use synthetic users

We treat synthetic respondents as a drafting tool, not a data source. That matches NN/g’s guidance to treat synthetic findings as a starting point, not as evidence of what users think, say or do ^[1]. In practice that means a few simple rules.

• Use them early. Draft hypotheses, sharpen interview guides and spot confusing survey wording before you recruit anyone.
• Use them for unfamiliar domains. NN/g notes they could help teams starting work in a completely new field ^[3].
• Never use them to validate a concept. Their tendency to praise ideas produces false confidence ^[3].
• Always follow up with real people. Small rounds of about five users find most usability problems, and several small rounds beat one big study ^[4].
• Write down what the synthetic panel predicted, then compare it with what real users said. The gaps are your best insights.

FAQ: synthetic users in UX research

Can synthetic users replace user interviews? No. NN/g’s 2026 study shows they can match the most common answer but miss the variety and subgroup differences real people show ^[1], and its earlier work advises against using them as a replacement ^[3].

What are synthetic users good for? Generating hypotheses, checking broad patterns and preparing research in a new domain ^{[1, 3]}. Think of them as a rehearsal, not the performance.

How many real users do we need instead? For usability testing, Jakob Nielsen’s long-standing guidance is about five users per round, repeated as the design improves ^[4]. Interviews for discovery may need more, depending on how many distinct audiences you serve.