Persona Prompts Change More Than Tone: Evidence from 750 Grounded AI Responses

At a glance

Research snapshot

Study overview

Executive summary

Persona prompting is often treated as a superficial prompt-engineering choice.

A researcher may ask an AI system to “act as a CMO,” “act as an SEO Manager” or “act as a Founder” and receive noticeably different responses. But that observation alone tells us very little.

The obvious alternative explanation is that the system simply repeats the language supplied in the persona prompt.

We designed a matched experiment to examine that problem more deeply.

The study began with 150 neutral business scenarios. Each scenario was rendered under five complete persona-conditioning bundles:

• CMO
• VP of Product Marketing
• SEO Manager
• Demand Generation Director
• Founder/CEO

That produced 750 historical grounded responses from Gemini.

Rather than analysing the responses once and stopping, we subjected the experiment to four successive analytical tests.

1. First

   We audited whether the historical run was sufficiently stable to analyse at all.

2. Second

   We removed direct persona language from the generated answers to test whether apparent persona differences were mostly prompt echo.

3. Third

   We inspected the search queries generated during grounded answering to determine whether persona-associated differentiation appeared before the final prose.

4. Fourth

   We tested whether the differentiation was concentrated in certain tasks or remained broadly present across different forms of business work.

The result is a more nuanced picture than either “personas do not matter” or “personas behave like real executives.”

The five persona conditions were associated with persistent differences in both answer construction and search formulation. Those differences survived direct persona-language removal and appeared across all five task families.

But the study also defines important boundaries.

We cannot yet establish whether the personas produced materially different factual claims, recommendations or decisions. We cannot identify publisher-level source differences from the stored grounding data. And because persona order was fixed during the historical run, we cannot isolate persona from collection-position effects.

The strongest conclusion is therefore narrower:

Persona conditioning should be treated as a meaningful experimental variable in grounded-AI research, not merely as a change in tone.

Research question

Why we ran this study

Persona prompts are increasingly used in AI research, content analysis, customer research and decision support.

A common approach is to ask the same AI system to respond from different roles:

“Answer as a CMO.”

“Answer as an SEO Manager.”

“Answer as a Founder.”

The assumption is that these prompts expose different perspectives.

But there are at least three ways this could be misleading.

• Prompt vocabulary

  First, the system could simply repeat vocabulary from the persona prompt.

• Underlying process

  Second, the final answer could sound different while relying on essentially the same underlying information-gathering process.

• Task sensitivity

  Third, persona differentiation could appear only for certain kinds of questions and disappear for others.

We wanted to separate these possibilities.

Instead of asking whether persona answers merely look different, the study progressively tested:

1. Historical stability

   Whether the historical experiment was trustworthy enough to analyse.

2. Direct persona language

   Whether differences remained after direct persona language was removed.

3. Grounded search queries

   Whether differences were visible in the model's grounded search queries.

4. Task families

   Whether those differences persisted across different kinds of tasks.

The resulting analysis turns persona prompting from a stylistic question into an experimental-design question.

For earlier Kojable work on role framing and persona effects, see How AI Role Prompts Change the Decision Lens, AI Persona Effects Beyond Prompt Echo, Persona Conditioning in AEO, and Persona Similarity in AI Responses.

Finding 1

The historical run was usable—but only for exploratory claims

Before studying persona effects, we audited whether the underlying run was trustworthy.

Of the 750 historical responses, 749 passed the substantive-response quality screen.

One response—N093::SEO Manager—contained raw tool-call syntax rather than a substantive answer and was excluded from the new substantive cohort.

The important question was whether removing that response materially changed the existing matched results.

It did not.

A major matched effect-size statistic for the first historical lexical analysis changed from:

0.4705 → 0.4738

and the second from:

0.4534 → 0.4518.

No effect direction or corrected pairwise decision changed.

That means the main historical persona-separation patterns were not being produced by the malformed response.

We also examined the collection window for evidence of substantial drift.

The full run lasted approximately 16.6 hours. Retries and high-latency responses became somewhat more common later in the run, but the predefined within-cluster drift screens did not reach the level required to classify the experiment as unstable.

The final quality decision was therefore:

Suitable with limitations.

The most important limitation is structural.

The five personas were collected in the same order for every scenario:

CMO → VP of Product Marketing → SEO Manager → Demand Generation Director → Founder/CEO

Persona condition and within-scenario collection position are therefore perfectly confounded.

If the third answer in every block differs systematically from the first answer, the historical data cannot tell us whether that occurred because of the persona instruction, the collection position, or some combination of the two.

No regression can recover that missing counterfactual after the fact.

For that reason, this report uses language such as:

“persona-conditioned outputs were associated with…”

rather than:

“the persona instruction caused…”

The experiment supports exploratory matched comparisons, not causal persona claims.

Evidence summary

What the study establishes

Taken together, the analyses support four main conclusions.

First, the historical matched run is sufficiently robust for exploratory analysis after removing one malformed response, although its fixed collection order prevents causal persona attribution.

Second, persona-associated answer differentiation survives removal of directly injected persona language.

Third, the differentiation is visible in grounded search-query formulation as well as in the final answers.

Fourth, it remains broadly persistent across five different kinds of business task.

These findings support the idea that persona choice should be treated as a meaningful experimental variable when designing grounded-AI research.

They do not establish that the persona conditions accurately simulate real executives.

Substantive status

The unresolved question: does different mean substantively different?

After establishing persistent representational differences, we attempted to measure a harder layer:

Do the persona conditions make materially different factual claims, recommendations, decisions and risk assessments?

The first deterministic extraction pipeline failed its measurement-validity check.

It produced candidate text but accepted no substantive units under its original rules.

We therefore did not interpret that result as evidence that the persona answers were substantively equivalent.

Instead, we created a separate measurement-validation stage.

A frozen 50-response calibration set has now been prepared across all five personas and all five task types.

The next step requires:

1. Two independent human reviewers
2. Separate adjudication
3. Validation of factual-claim extraction
4. Validation of recommendation extraction
5. Validation of risk extraction
6. Validation of decision classification
7. Validation of proposition-equivalence matching

Until that calibration is complete, the correct substantive conclusion is:

Not established.

This matters because it separates two questions that are often collapsed in AI evaluation:

Do outputs differ?

and:

Do those differences matter?

This study answers the first much more clearly than the second.

Interpretation

Implications for AI research

The practical implication is not that every company should immediately create five personas.

It is that persona choice should not be treated as decorative prompt wording.

If the same underlying scenario produces different grounded searches and persistently different answer structures under different persona conditions, then evaluating an AI system through only one persona can under-sample the range of responses the system is capable of producing.

That matters for research designs involving:

• AI visibility
• Answer Alignment
• Buyer-question testing
• AI search evaluation
• Competitive research
• Decision-support systems
• Prompt-panel design

A single generic prompt may answer:

“What does the model say?”

A persona panel begins to answer:

“How does the answer space change when the decision context changes?”

That is a different research problem.

The results also suggest that persona-conditioned research should be evaluated at more than the final prose layer.

Search formulation itself changed substantially.

For grounded AI systems, that means evaluation can potentially examine:

prompt → query formulation → evidence retrieval → answer

rather than treating the answer as a black box.

This study could measure the first, second and fourth parts of that pathway.

Publisher-level evidence retrieval remains an unresolved layer.

Implications for AI Answer Alignment

For companies monitoring how AI systems represent them, the findings raise an important methodological question.

A company may appear correctly represented under one broad query but differently represented when the same underlying issue is approached from:

• Strategic risk
• Technical implementation
• Revenue generation
• Positioning
• Executive prioritisation

If persona-conditioned contexts systematically alter both search formulation and answer construction, a single generic question may not fully represent the decision environments in which buyers or stakeholders encounter the company.

This does not mean research programmes should create arbitrary personas simply to generate more outputs.

It means persona design should be deliberate, tested and tied to distinct decision contexts.

The next question is whether those contexts produce genuinely different substantive conclusions.

That is where the current research programme is heading.

Next phase

The next experiment

There are two clear next steps.

The first is already underway:

Validate substantive divergence through independent human calibration.

That will test whether the five persona conditions contribute genuinely different:

• Facts
• Recommendations
• Decisions
• Risks

The second is experimental redesign.

A confirmatory collection should randomize persona order, record generation controls and collect repeated outputs per persona/scenario.

That would allow the research programme to distinguish:

• Persona-condition effects
• Generation variability
• Collection-position effects
• True within-persona repeatability

Only then should stronger causal or equivalence claims be considered.

Conclusion

Conclusion

Persona prompting is often discussed as if it were mainly a question of tone.

This experiment suggests a more consequential interpretation.

Across 150 matched scenarios, the five persona conditions were associated with persistent differences in the model's final answers.

Those differences survived direct persona-language removal.

They were also visible in the grounded search queries the model formulated.

And they remained broadly present across five different task families.

At the same time, the study draws a clear boundary around what has not yet been established.

We cannot isolate persona from historical collection position.

We cannot identify publisher-level source differences.

And we cannot yet say whether the observed response diversity corresponds to materially different facts, recommendations or decisions.

The strongest current conclusion is therefore not:

Personas make AI answers better.

It is:

Persona conditioning changes the observed answer space enough that it should be treated as an experimental design variable—not merely as a stylistic prompt choice.

The next phase is to determine whether that representational diversity translates into substantive decision value.

Research context

Related Kojable research

• How Persona Conditioning Changes AEO Answers

  The earlier 750-response matched AEO analysis.

• Beyond Persona Labels: Do AI Personas Still Differ After Explicit Cues Are Removed?

  Cue-ablation research on a separate finance-persona corpus.

• When Do Personas Really Change AI Responses?

  Earlier persona-similarity research.

• Do AI Role Prompts Change the Decision Lens?

  Role-framing research using a separate marketing corpus.

• How Predictable Are Gemini’s Fan-Out Queries?

  Related research on prompt wording and grounded Gemini search queries.

• AI Answer and Source Research

  Browse the Kojable Research index.