Your Next User Test Might Not Include a Single Human

Imagine your product is five days from launch.

The navigation still feels awkward. Nobody agrees on the headline. One person thinks the checkout flow is obvious; another thinks it is confusing. The team needs evidence, but recruiting participants will take longer than the deadline allows.

Then someone makes an unusual suggestion: "Why don't we test it with people who don't exist?"

That is no longer a science-fiction question.

AI systems can now generate synthetic users – simulated participants with assigned ages, jobs, goals, anxieties, budgets, and buying habits. Some can inspect a live website, click through a product flow, respond to interview questions, and explain what they supposedly thought at every step.

Minutes later, the team receives a professional-looking report filled with friction points, quotes, sentiment labels, and recommendations.

It feels like research. It looks like research. But is it evidence?

That question matters because synthetic users may become one of the most useful – and most easily misunderstood – applications of generative AI.

Meet the customers who do not exist

A synthetic user is typically a large language model instructed to act like a particular person or customer segment.

The basic version starts with a prompt: "You are a first-time buyer, 35 years old, working in finance, cautious about subscriptions, and shopping on a phone."

The model then reacts to a product from that perspective.

More advanced systems add survey results, customer interviews, product documentation, analytics, or browsing tools. Platforms can generate an entire panel of simulated respondents and send them through a website or prototype. Vendors such as Delve AI combine this process with journey maps, simulated interviews, heatmaps, and usability reports.

The technology is impressive. The terminology is also misleading.

The model has no rent to pay, no impatient manager, no crying child in the next room, and no fear of wasting its own money. It does not experience confusion or frustration. It predicts language that would plausibly describe those experiences.

That does not make the output worthless. It makes the output a simulation.

Why teams will find this impossible to resist

Traditional user research contains friction before the research even begins.

Participants must be found, screened, scheduled, compensated, and interviewed. Studies may require specialist staff and several rounds of coordination. When a team is choosing between two early concepts, that investment can feel disproportionate.

Synthetic participants change the economics of asking a question. A product team can run the same task repeatedly, explore several customer profiles, and test revised copy almost immediately.

They are especially useful for detecting problems such as the following:

• An unclear value proposition

• A button whose purpose is ambiguous

• Important pricing information buried too deeply

• Filters that ask for data without explaining why

• Contradictory instructions or broken paths

• Onboarding language that assumes too much knowledge

These problems are real, but they are not always profound. Many are structured failures of clarity and consistency.

That suggests a useful comparison: synthetic testing could become the spell-checker of customer experience.

A spell-checker catches errors quickly and cheaply. It can scan every revision without becoming tired. But nobody would ask a spell-checker whether a novel is emotionally honest.

Synthetic users deserve the same boundary. They may be excellent at finding visible mistakes and poor at explaining what those mistakes mean in a human life.

The research is promising – and awkward

Evidence from behavioural research shows why simple answers are dangerous.

In a 2023 paper presented at the International Conference on Machine Learning, Gati Aher, Rosa Arriaga, and Adam Tauman Kalai used language models to simulate participants in several famous experiments. The models reproduced findings associated with the ultimatum game, garden-path sentences, and the Milgram experiment.

Yet another experiment exposed what the researchers called a hyper-accuracy distortion. The simulated crowd performed too well on a task where ordinary human error was part of the result. Read the ICML paper.

In other words, the model sometimes reproduced the pattern while missing the mess.

Another study found that GPT-3 could reflect meaningful differences between human subpopulations when it was conditioned on thousands of demographic backstories drawn from real surveys. The researchers called the effect algorithmic fidelity. Read the study in Political Analysis.

This is the strongest argument for synthetic users. Models can sometimes reproduce recognisable human patterns when they receive enough relevant context.

But recognisable is not interchangeable.

A 2024 study led by James Bisbee compared ChatGPT-generated survey responses with human survey data. Some averages looked similar. The synthetic responses, however, showed less variation, changed when the wording of the prompt changed, and shifted over time even when the same prompt was reused. The authors warned that the results were not reliable for statistical inference. Read the study.

That missing variation is crucial. In user research, the awkward outlier may contain the discovery.

The most dangerous output is the polished one

A rough AI answer naturally invites scepticism. A beautifully designed research dashboard does the opposite.

Imagine seeing 100 simulated interviews summarised into exact percentages, recurring themes, emotional labels, and prioritised fixes. The presentation creates a sense of measurement. Even disagreements between synthetic respondents can make the panel feel authentic.

But every result depends on hidden choices:

• Which model generated the participants?

• What information shaped their personas?

• Which populations were missing from the training data?

• How were the tasks and questions phrased?

• Would another model produce the same conclusion?

• Has any prediction been compared with real behaviour?

A system can produce precise output from uncertain assumptions.

This is why 'bias-free participants' is the wrong sales pitch. Synthetic users avoid some human research problems: fatigue, cancellations, scheduling, and the desire to please an interviewer. They introduce different problems: model bias, prompt sensitivity, compressed variation, and confident explanations generated after the fact.

AI does not remove bias from research. It relocates it.

AI first. Humans second

The best argument for synthetic users is not that companies need fewer people. It is that access to people should be spent more intelligently.

Here is a practical four-stage process:

1. Let AI find the obvious

Use synthetic participants and automated checks on early concepts. Look for confusing copy, inconsistent navigation, missing information, predictable objections, broken flows, and simple accessibility problems.

Fix those issues before asking a person to spend time finding them.

2. Let people reveal reality

Test the improved version with real members of the intended audience. Watch for hesitation, emotion, social pressure, unusual environments, and improvised workarounds.

Do not only ask, "Did the design work?" Ask, What did the simulation fail to imagine?

3. Compare prediction with behaviour

Measure the calibration gap. Where did the synthetic panel identify genuine friction? Where did it exaggerate a concern? Which customers did it represent badly? Which findings appeared only when real consequences were involved?

Without calibration, a synthetic user is a persuasive stranger. With calibration, it can become a limited instrument whose strengths and failure modes are understood.

4. Match the evidence to the risk

Use synthetic feedback more freely for cheap, reversible choices. Demand stronger human evidence as the consequences rise.

Changing a landing-page headline is not the same as redesigning a healthcare workflow. Testing a product description is not the same as deciding that an underserved community has no unmet needs.

The higher the stakes, the less simulation should count as proof.

The rule worth remembering

Synthetic users are useful when the cost of asking people is high, and the cost of a wrong prediction is low.

Human research becomes essential when the decision is consequential, the behaviour is emotionally or socially complex, the audience is poorly represented in existing data, or the team needs to discover something genuinely new.

That creates a simple division of labour:

Use AI for speed, breadth, repetition, and hypothesis generation.

Use people for truth, context, surprise, and consequence.

As AI makes plausible answers nearly free, the value of judgement will rise. Product teams will not struggle to generate feedback. They will struggle to decide which feedback deserves belief.

The winners will not be the companies that use synthetic personas to avoid their customers. They will be the companies that use simulations to arrive at human conversations with sharper questions.

Because the purpose of research was never to collect the largest number of answers.

It was to encounter reality before reality became expensive.

If you want clear, evidence-based analysis of AI tools without the hype, subscribe to my Vocal profile. Then tell me in the comments: What decision would you never trust to a synthetic user?