Kojable research
How Many AI Prompts Do You Really Need to Track? A 180-Prompt Gemini Study
AEO teams often monitor large prompt libraries because small changes in wording, role, intent, geography, or integration can appear commercially important. Kojable tested whether that duplication also appears in the overall answers Gemini produces for related B2B finance questions.
Similar prompts produced strongly similar overall answers in this study, supporting representative prompt monitoring when teams also validate the brand-level outcomes that matter.
Quick answer
Cluster related prompts before treating each one as a new monitoring job
Prompt clustering groups AI queries that express similar meanings. In this experiment, semantic similarity between two prompts had a very strong relationship with semantic similarity between the Gemini answers they produced.
The result supports representative prompt monitoring, not blind consolidation. It does not prove that brand mentions, citations, recommendations, or vendor order remain identical across every variation. Those outcomes need their own checks.
Research question
When do prompt variations deserve separate AEO monitoring?
A company may want to follow questions such as “What is the best cash flow software for a SaaS business?”, “Which cash flow tools integrate with NetSuite?”, or “How should a CFO forecast free cash flow?”. Each differs by context, specificity, and likely commercial value.
If even nearby prompts create meaningfully different answers, a team must maintain a large prompt universe. If related prompts consistently produce related answers, it can organise monitoring around a smaller set of representative prompts and use selected variations to validate what is commercially important.
Experiment at a glance
A pairwise study of 180 B2B finance prompts
The prompt set covered free cash flow, B2B payment automation, and transaction fraud detection. Each topic contributed 60 prompts, creating both close variations and clearly different information needs.
Total prompts
60 prompts in each topic.
Topic clusters
Free cash flow, payments, and fraud detection.
Unique pairs
Every unordered pair among 180 prompts.
Within-topic pairs
Comparisons between prompts in the same topic.
Cross-topic pairs
Comparisons between prompts in different topics.
Robustness runs
Prompt-level bootstrap iterations and Mantel permutations.
Prompt variation and answer generation
The prompts varied like real buyer research, not just by synonym swaps
The set varied persona, industry, geography, integration requirements, intent, and prompt form. It included CFOs, controllers, treasury managers, and accounts-receivable managers; SaaS, fintech, and payments contexts; and Ireland, the United Kingdom, and the United States.
Prompts also included integrations such as NetSuite, Stripe, and SAP; informational, educational, commercial, and
transactional intent; and questions, commands, and short search-style phrases. Each prompt was submitted to
gemini-3-flash with Google Search grounding enabled. Enabling grounding does not necessarily mean a live
search ran for every individual prompt.
Persona
Questions reflected roles with distinct finance responsibilities and information needs.
Industry and geography
Contexts included SaaS, fintech, payments, Ireland, the United Kingdom, and the United States.
Integration requirements
Some variations specified systems such as NetSuite, Stripe, or SAP.
Intent and form
The set mixed research, learning, buying, and task-oriented prompts in different query forms.
Semantic-similarity methodology
Meaning matters more than word overlap
A keyword comparison would miss related phrases with different vocabulary, such as “working capital platform” and
“cash management software”. The study therefore converted every prompt and every answer into an embedding using
gemini-embedding-001.
An embedding represents text as a numerical vector. Cosine similarity compares the direction of two vectors: higher values mean the texts sit closer together in the embedding space. It is a measure of similarity, not a probability that two texts mean exactly the same thing.
For each unordered pair, the analysis recorded prompt similarity on one dimension and the similarity of the two corresponding answers on the other. With 180 prompts, the calculation is 180 × 179 ÷ 2 = 16,110 pairs.
Findings
Prompt structure was strongly reflected in the overall answers
Similar prompts produced similar overall answers
The Pearson correlation shows a very strong positive linear relationship in this dataset.
Most aligned variation was substantial
Descriptively, 77.1% of pairwise response-similarity variation aligned with prompt similarity.
Topic separation was large
Same-topic and cross-topic answer distributions were separated by more than one pooled standard deviation.
Finding 1
Similar prompts produced similar overall answers
The Pearson correlation between prompt similarity and response similarity was r = 0.878. In plain language, prompts that were semantically closer tended to receive answers that were semantically closer too. Prompt wording was not a minor signal in this experiment.
The squared correlation was r2 = 0.771. That is a descriptive indication that about 77.1% of variation in pairwise response similarity aligned with variation in pairwise prompt similarity. It does not establish causation or prove the same relationship for other models, industries, or time periods.
Finding 2
Gemini maintained topic boundaries
Within-topic pairs had a mean answer similarity of 0.664. Cross-topic pairs had a mean of 0.569. The absolute gap was 0.095, making within-topic answers approximately 16.7% more similar when the cross-topic mean is used as the baseline.
This control matters. A high overall correlation would be less useful if every answer was simply similar in tone or format. The gap indicates that the embedding method detected meaningful topic structure across the responses.
Finding 3
The separation was large, not merely detectable
The reported standardised effect size was Cohen’s d = 1.27. Cohen’s d expresses the difference between group means in standard-deviation units. On commonly used conventions, this would be called large, although those thresholds are guidelines rather than universal rules.
Here the practical reading is direct: same-topic and different-topic response distributions were separated by considerably more than one pooled standard deviation.
Robustness checks
Why a naive significance test was misleading
A conventional t-test across all 16,110 pairs produced an extremely large statistic and a near-zero p-value. But the pairs were not independent: each prompt appeared in 179 comparisons. An unusual answer therefore affected every pair involving that prompt.
Treating those pairs as independent would create pseudoreplication, exaggerating the effective sample size and statistical power. The prompt, rather than each pair, is the experimental unit.
-
Stratified prompt-level bootstrap
Across 2,000 iterations, the study resampled original prompts within each topic and reconstructed comparisons, preserving more of the dependency structure.
-
Mantel permutation test
Across 2,000 permutations, the full prompt-similarity and response-similarity matrices were compared. No random permutation equalled or exceeded the observed relationship.
-
Empirical result
The reported Mantel result was p < 0.001, adding evidence that the alignment was unlikely to reflect an arbitrary matching of the two matrices.
What the experiment demonstrates
Four conclusions for AEO measurement
Prompt wording has measurable structure
Persona, intent, integration, industry, and phrasing form tighter and looser semantic groups.
Overall answer meaning followed that structure
The relationship makes prompt clustering a practical measurement technique, not only an organisational convenience.
Every paraphrase may not need primary monitoring
Clustering can reduce duplicated API calls and analyst review of similar broad answer behaviour.
A representative prompt is useful, not automatically sufficient
Overall similarity can coexist with material differences in mentions, citations, recommendations, or factual accuracy.
Practical workflow
A seven-step prompt-clustering workflow
Treat clusters as working measurement groups, not as a permission to discard commercially meaningful variations. The workflow below keeps a seed prompt for efficient monitoring and a validation set for the details that can change a business decision.
Build the real prompt universe
Start with customer, prospect, and evaluator questions across role, industry, use case, maturity, geography, integration, buying intent, competitor, and desired output. Do not create variations solely by swapping synonyms.
Generate prompt embeddings
Use one consistent embedding model and configuration. Record the model, task type, output dimensionality, normalisation method, and generation date; embeddings from different configurations should not be assumed to be comparable.
Cluster prompts by semantic similarity
Group prompts that are sufficiently close in embedding space. A cosine threshold such as
0.75is only an initial operating heuristic, not a universal boundary; test it against human judgement, cluster size, intent purity, response similarity, and brand-outcome stability.Choose a seed prompt and validation prompts
Select a central seed for each cluster, then retain important commercial, specific, persona, competitor, regional, and integration variations as validation prompts.
Measure brand-level outcomes
Track brand mention rate, citation rate, recommendation rate, share of answer, competitor co-mentions, vendor order, source overlap, sentiment, entity accuracy, missing attributes, and retrieval volatility.
Compare the seed with its cluster
Approve a seed only when high overall similarity is joined by sufficiently stable commercial outcomes. The acceptance rule should match the business risk.
Retest over 7, 14, and 30 days
Watch short-term movement, changes after content or source work, and longer trends. A cluster that is consistent today can split after a model or retrieval update.
Share of answer is the percentage of AI-generated answers in a defined query set that mention, cite, recommend, or rely on a brand relative to its competitors.
Illustrative operating scenario
From 200 prompts to 20 primary clusters
Consider a team with 200 prompt variations that identifies 20 meaningful clusters. It could monitor 20 seed prompts, use a rotating validation sample, and add prompts only for high-risk clusters. That is a 90% reduction in primary prompt volume and ten times fewer primary model calls.
This is an illustrative operating scenario, not an efficiency result measured directly by the experiment. The right consolidation rate depends on the category, platform, cluster threshold, and stability of brand-level outcomes.
Kojable interpretation
Similarity shows what is happening; evidence helps explain why
The useful first conclusion is that teams may monitor fewer primary prompts. The deeper question is why related prompts generate related answers. Possible mechanisms include similar underlying intent, related fan-out searches, recurring domains in retrieval, reused source passages, stable answer templates, or concentrated public evidence.
This study measures the relationship between prompts and outputs; it does not isolate which mechanism created it. Kojable’s evidence-backed Answer Intelligence approach moves from Monitor to Diagnose, Improve, and Verify: understanding the sources and information gaps shaping AI answers, then testing whether the relevant changes improved them. Similarity is a useful signal, but it is not a causal explanation on its own.
Next research layer
What would make prompt clustering more decision-ready?
Brand-state consistency
Classify whether a target brand is absent, mentioned, cited, recommended, or ranked first across a cluster.
Citation-source overlap
Compare cited domains and URLs to see whether similar answers rely on the same information path.
Fan-out query similarity
Capture model-generated searches when grounding occurs to test whether related prompts produce related retrieval plans.
Temporal stability
Repeat after 7, 14, and 30 days, a model update, or a meaningful change in public source evidence.
Cross-platform replication
Test whether a consolidation pattern carries across answer engines rather than assuming one platform represents all of them.
Limitations
What this study does not prove
- One seed prompt can always replace every variation in its cluster.
- Brand mentions, vendor order, or cited sources remain stable across related prompts.
- A cosine threshold of
0.75is universally optimal. - The observed
r = 0.878will remain unchanged after model updates. - Results generalise beyond B2B finance or to other answer engines.
- Google Search grounding ran for every submitted prompt.
- Semantic similarity alone is an adequate AEO success metric.
These limits do not remove the finding. They define the next measurement problem: validate the commercial and evidence-level outcomes inside each cluster instead of assuming that overall answer similarity settles them.
Bottom line
Cluster first, then validate what affects the business
In this 180-prompt Gemini experiment, prompt similarity and overall response similarity moved together strongly: r = 0.878. Same-topic answers averaged 0.664 similarity versus 0.569 for cross-topic answers, and the reported separation was Cohen’s d = 1.27.
Cluster prompts first. Monitor representative seeds. Validate brand-level outcomes. Investigate the sources and retrieval mechanisms behind any differences.
FAQ
Questions about AI prompt similarity
What is AI prompt clustering?
It groups prompts with similar meanings or user intents so teams can organise monitoring around representative seed prompts.
How many prompts were included in the study?
The study included 180 prompts: 60 prompts in each of three B2B finance topics.
How many prompt pairs were analysed?
The 180 prompts produced 16,110 unique unordered pairs: 5,310 within-topic pairs and 10,800 cross-topic pairs.
What does a Pearson correlation of 0.878 mean?
It means prompt similarity and response similarity had a very strong positive linear relationship in this dataset.
Does the study prove one prompt can represent an entire cluster?
No. A seed can monitor general answer behaviour, but the study did not prove that mentions, citations, or recommendation order remain identical.
Why use embeddings instead of keyword overlap?
Embeddings represent semantic meaning, so related phrases with different vocabulary can still receive a higher similarity score.
Why was the naive t-test rejected?
Each prompt appeared in 179 pairs, so the observations were not independent. Treating them as independent would exaggerate significance.
What should AEO teams measure after clustering?
Measure mentions, citations, recommendations, share of answer, competitor presence, source overlap, sentiment, entity accuracy, and answer volatility.
Is a cosine-similarity threshold of 0.75 always appropriate?
No. It is a starting heuristic only; the right threshold depends on the embedding model, category, prompt diversity, and required outcome stability.
How does Kojable use this analysis?
Kojable treats answer-level measurement as the start of a diagnosis: identifying what may shape an answer, what evidence should change, and what to retest.
Next step
Move from prompt volume to evidence-backed answer intelligence
See how Kojable monitors AI representation, diagnoses the evidence shaping the result, guides improvement, and verifies what changed.