Kojable research · Cross-Provider AI Query Fan-Out Study · Companion analysis
Similar AI Search Plans, Different Source Pools
Why semantic query overlap did not reliably translate into the same retrieved or cited evidence
Two AI systems can search for the same thing without surfacing the same evidence.
That distinction matters because fan-out analysis is often interpreted as a bridge from what an AI system searches for to which sources a company needs to influence or appear in.
Our cross-provider query fan-out experiment gives us a way to test that bridge.
Claude, Gemini, OpenAI and Perplexity received the same ten designed B2B buyer questions and the same candidate-query scaffold. We measured how similar their observable search plans were semantically, then compared the URL and domain pools exposed by their search and citation traces.
The result was not a simple pipeline from:
similar query plan → similar source pool
Among the three providers with defensibly comparable destination URLs—Claude, OpenAI and Perplexity—mean retrieved-URL Jaccard overlap was only 0.006 to 0.037 across provider pairs. Mean retrieved-domain overlap was 0.040 to 0.084. Final cited-source overlap was similarly small.
A particularly clear example came from P10. OpenAI and Perplexity had a semantic-cluster Jaccard of 1.00 for that buyer question: their observable query plans covered the same semantic cluster set. Yet their retrieved-URL Jaccard was 0.00 and their cited-URL Jaccard was also 0.00.
Key finding
Similar observable search plans did not reliably imply highly overlapping observable evidence pools. Query-plan similarity and source identity are related layers of AI search, but they are not interchangeable measurements.
Qualification: Source-pool comparisons are constrained by provider telemetry. Result-pool sizes differ, OpenAI and Perplexity expose batch-level result linkage while Claude exposes query-level linkage, and Gemini’s exported destination URLs remain unresolved provider wrappers. The study therefore describes observable source pools rather than hidden backend retrieval, and it does not measure retrieval efficiency.
Read the flagship cross-provider study
- AI search plans
- Source overlap
- AEO measurement
- 11 min read
Answer first
Did similar AI search plans surface the same sources?
Direct answer
Not reliably in this frozen benchmark.
For the comparable provider pairs:
| Provider pair | Mean semantic-cluster Jaccard | Retrieved URL Jaccard | Retrieved domain Jaccard | Cited URL Jaccard | Cited domain Jaccard |
|---|---|---|---|---|---|
| Claude–Perplexity | 0.262 | 0.037 | 0.084 | 0.009 | 0.017 |
| OpenAI–Claude | 0.295 | 0.014 | 0.045 | 0.018 | 0.043 |
| OpenAI–Perplexity | 0.593 | 0.006 | 0.040 | 0.012 | 0.018 |
OpenAI and Perplexity had by far the highest mean semantic overlap of these three pairs, but they had the lowest mean retrieved-URL overlap.
Claude and Perplexity had lower semantic overlap but the highest retrieved URL and domain overlap of the three pairs.
That does not mean semantic similarity is irrelevant. It means the mapping from query plan to source identity is not one-to-one.
Measurement distinction
1. Query similarity and source similarity measure different things
A semantic query cluster represents an information need.
A URL is a specific piece of evidence.
Those are different levels of the retrieval process.
Two providers can agree that a buyer question requires research into measurement, pricing, implementation, alternatives, evidence quality or vendor evaluation while still selecting different pages, publishers and domains to satisfy those needs.
Different search systems may rank different results for related queries, use different indexes or retrieval layers, issue different surface forms, retrieve at different depths, apply different filtering rules, or cite only a subset of retrieved evidence.
So semantic overlap should not be expected to produce identical source sets.
The empirical question is:
As search plans become more semantically similar, do their observed source pools become meaningfully more similar too?
Finding 1
2. Observable source overlap was very low
Across the three comparable provider pairs, mean retrieved-URL overlap never exceeded 0.037.
Mean retrieved-domain overlap was somewhat higher, but still low:
- Claude–Perplexity: 0.084
- OpenAI–Claude: 0.045
- OpenAI–Perplexity: 0.040
Final cited-source overlap was also small.
Mean cited-URL Jaccard ranged from 0.009 for Claude–Perplexity to 0.018 for OpenAI–Claude.
Mean cited-domain Jaccard ranged from approximately 0.017 to 0.043.
These values are descriptive averages across the ten buyer questions. They should not be read as stable long-run provider constants.
But the absolute scale is informative:
The systems repeatedly surfaced largely non-identical URL sets for the same buyer question.
Illustrative case
3. The P10 case shows why the distinction matters
P10 asked a B2B vendor-evaluation question.
For OpenAI and Perplexity, the query-level measurements were unusually aligned:
- candidate-selection Jaccard: 1.00
- semantic-cluster Jaccard: 1.00
- query-family Jaccard: 1.00
On those measures, the observable search plans were as similar as this analysis could record.
But the evidence pools still diverged:
- retrieved-URL Jaccard: 0.00
- retrieved-domain Jaccard: 0.049
- cited-URL Jaccard: 0.00
- cited-domain Jaccard: 0.00
This is not proof that semantic query planning and source selection are unrelated.
It is a concrete example of why semantic consensus cannot be treated as source consensus.
The same information-needs map can be satisfied by different evidence.
Finding 2
4. Across prompts, retrieved-source overlap did not rise with semantic overlap
The study tested the relationship directly across 30 comparable prompt × provider-pair observations.
For semantic-cluster overlap versus retrieved-URL overlap:
- Spearman ρ = −0.108
- prompt-cluster bootstrap interval: [−0.498, 0.251]
For retrieved-domain overlap:
- ρ = −0.184
- interval: [−0.515, 0.137]
Both intervals include zero and are broad.
The data therefore do not support a reliable positive relationship between semantic-query similarity and retrieved URL/domain similarity in this sample.
This is not evidence of equivalence or independence.
The conclusion is narrower:
More similar semantic query plans did not reliably correspond to more similar retrieved source pools in this experiment.
Finding 3
5. Cited-source overlap showed a different, but still limited, pattern
The citation layer was slightly different.
Semantic-cluster overlap versus cited-URL overlap had:
- ρ = 0.288
- bootstrap interval: [0.049, 0.547]
Semantic overlap versus cited-domain overlap had:
- ρ = 0.215
- interval: [−0.098, 0.538]
The cited-URL association is directionally positive and its exploratory bootstrap interval excludes zero.
That is worth reporting, but it should not be overstated.
The absolute cited-URL Jaccard values remained very small, there are only 30 comparable pair-prompt rows, and the analysis is exploratory.
A useful interpretation is:
Query-plan similarity may have had more correspondence with final cited-URL overlap than with the broader retrieved pool, but absolute overlap remained low.
Telemetry boundary
6. Why Gemini is excluded from URL-overlap interpretation
The full fan-out experiment includes Gemini.
The source-identity overlap analysis does not.
Gemini’s exported destination records remain unresolved vertexaisearch.cloud.google.com wrappers in the frozen data.
That creates a visible artifact: Gemini frequently appears to have only one domain in a run because the exported wrapper domain is being observed instead of the true destination domains.
If those wrappers were compared directly with Claude, OpenAI and Perplexity destinations, the result would mechanically look like zero URL/domain overlap.
That would be a telemetry difference, not a defensible source-divergence result.
So Gemini’s raw zero-overlap values should not be interpreted as evidence that Gemini searched an entirely different web.
Until redirects are resolved, Gemini is excluded from source-identity overlap claims.
Measurement boundary
7. Result-pool depth complicates raw overlap
The providers exposed very different result-pool sizes.
For example, in P01:
- Claude exposed 81 unique URLs
- OpenAI exposed 89
- Perplexity exposed 45
Other prompts also showed substantial variation.
A raw Jaccard score depends on both the number of shared items and the size of the union.
Providers with deeper or differently structured observable pools can therefore receive different overlap values partly because their exported sets have different sizes.
This is one reason the study avoids calling low source overlap a measure of retrieval quality or retrieval efficiency.
A future confirmatory experiment should compare matched top-k results, common depth cutoffs and rank-aware overlap.
Analysis unit
8. Query-result linkage is not equally observable across providers
The provider traces differ in granularity:
- Claude: direct query-level result linkage
- OpenAI: batch-level result linkage
- Perplexity: batch-level result linkage
- Gemini: run-level result linkage
This means we can compare prompt-level source pools for Claude, OpenAI and Perplexity reasonably well.
But we cannot always say:
“This exact generated query caused this exact URL to be returned”
for OpenAI or Perplexity.
The analysis is strongest at the level of:
provider × buyer-question source sets
not universal query-to-URL causal attribution.
Implications
What this means for AEO and GEO teams
A shared fan-out query does not imply a shared source opportunity
If multiple providers explore the same semantic need, the sources they surface may still differ.
So monitoring only query consensus can miss a second layer:
Which evidence environments satisfy that need for each provider?
The relevant object may be:
information need × provider × evidence surface
rather than one universal query-to-source map.
Optimize evidence coverage, not only query wording
The study suggests that companies should think beyond literal query matching.
A strong evidence strategy may require coverage across first-party documentation, specialist publishers, comparison pages, reviews, credible third-party analysis, technical evidence and category content.
The current experiment does not establish which source type causes better AI visibility.
Query consensus and citation consensus should be separate dashboard layers
A monitoring product should not collapse these into one “AI consensus” score.
Useful layers include:
- semantic query overlap;
- candidate-selection overlap;
- retrieved source overlap;
- cited source overlap.
They answer different questions.
Multiple evidence surfaces may be necessary
Low observed URL overlap suggests that visibility in one provider’s evidence pool may not automatically transfer to another.
This does not mean companies should create duplicate content for each provider.
It means evidence should be distributed across credible surfaces that can satisfy recurring buyer needs in multiple retrieval ecosystems.
That is a hypothesis for testing, not a causal result from this study.
Research relationship
How this differs from our citation-source-overlap research
Kojable has a separate report asking:
That study focuses directly on final citation-source overlap.
This companion asks:
Does similarity in the observable search plan predict similarity in the source pool?
The contribution is not another finding that source overlap is low.
It is the finding that query-plan similarity cannot safely be used as a proxy for source-pool similarity.
Constraints
Limitations
- Ten buyer questions: the association analysis has only 30 comparable prompt × provider-pair rows.
- One run per cell: source overlap may vary across repeated executions.
- Different result-pool sizes: raw Jaccard overlap is affected by set depth.
- Different telemetry granularity: query-to-result attribution is not uniform.
- Gemini destination wrappers: URL/domain overlap is not comparable and Gemini is excluded.
- Exploratory association analysis: bootstrap intervals are informative but not confirmatory.
- No rank-aware source overlap: the analysis treats observed sets, not ranked retrieval equivalence.
- No causal direction: query similarity may influence source similarity, both may reflect shared scaffolding, or provider retrieval systems may dominate the relationship.
Next experiment
What should be tested next
A stronger source-pool experiment would include:
- 30–50 or more prompts;
- multiple domains;
- repeated runs;
- seeded and unseeded fan-out conditions;
- resolved destination URLs for all providers;
- matched retrieval-depth comparisons;
- top-k and rank-biased overlap metrics;
- query-level result lineage where technically available;
- source quality and authority labels;
- time-separated collection.
The key research question becomes:
Conditional on two providers pursuing the same semantic retrieval need, how often do they surface the same evidence at a matched retrieval depth?
Conclusion
Conclusion
Similar AI search plans did not reliably produce similar observable source pools.
Across Claude, OpenAI and Perplexity:
- mean retrieved-URL Jaccard was only 0.006–0.037;
- mean retrieved-domain Jaccard was 0.040–0.084;
- mean cited-source overlap was also low.
Semantic-query overlap was not reliably positively associated with retrieved URL or domain overlap in this small sample.
The citation layer showed a somewhat different exploratory pattern: semantic overlap had a modest positive association with cited-URL overlap, even though absolute overlap remained small.
And one case made the distinction especially concrete:
OpenAI and Perplexity had identical semantic-cluster sets for P10, yet shared no retrieved URLs and no cited URLs.
Fan-out analysis answers one layer:
What information needs does the AI system explore?
Source analysis answers another:
Which evidence does it surface to satisfy those needs?
For AEO and GEO measurement, both layers need to be observed separately.
FAQ
Frequently asked questions
Did the providers retrieve the same URLs for the same buyer questions?
Usually not in the comparable source pools. Mean retrieved-URL Jaccard ranged from approximately 0.006 to 0.037 across Claude, OpenAI and Perplexity provider pairs.
Which pair had the highest semantic query overlap?
OpenAI–Perplexity, with mean semantic-cluster Jaccard of approximately 0.593.
Did OpenAI–Perplexity therefore have the highest source overlap?
No. It had the lowest mean retrieved-URL Jaccard of the three comparable provider pairs, approximately 0.006.
Does that prove query similarity has no effect on source selection?
No. The sample is small and the retrieved-source association intervals are broad.
Was there any relationship with citations?
Semantic overlap had a modest positive exploratory association with cited-URL overlap: Spearman ρ ≈ 0.288 with a prompt-cluster bootstrap interval of approximately [0.049, 0.547].
Why is Gemini excluded?
Its exported destinations remain unresolved provider redirect wrappers, so URL/domain identity is not comparable with the resolved destinations from Claude, OpenAI and Perplexity.
Does low source overlap mean one provider is better?
No. Source overlap is not a quality metric.
Should companies optimize for different URLs on every AI provider?
The study does not establish that strategy. It supports monitoring multiple evidence surfaces and testing whether credible source coverage transfers across provider ecosystems.
