Kojable case study · AI source discovery

Claude and Gemini Source Discovery in Vaccine Misinformation Monitoring

A September 2026 case study of URL overlap, exact-page matching and responsible human review.

Published By Kojable
System
AI-assisted source discovery and structured source assessment
Constraint
One completed full run per provider; conservative URL identity; no clinical validation of pages
Outcome
21 shared URL identities in a union of 499 (4.21% Jaccard overlap)

Author-reproduced first-party archive figures; not independently audited for this publication. Discovery is not a final-answer citation or an influence estimate.

Direct answer. In one completed source-discovery run per provider on 11 September 2026, Kojable recorded 455 Claude and 140 Gemini source observations across the same ten vaccine-misinformation monitoring tasks. After excluding unusable preferred URLs and conservatively normalising the remaining identifiers, the runs contained 412 and 108 distinct URL identities. Only 21 appeared in both sets, a 4.21% Jaccard overlap across 499 identities in the union. This is a finding about recorded discovery inventories—not medical accuracy, public influence or final-answer AI citations. Page identity, claim assessment and action eligibility still require separate checks.

Key findings

  • The discovered source pools differed.

    The run-wide preferred-URL comparison yielded 21 shared identities out of 499 in the union. This does not establish a stable provider preference or show that either model was more comprehensive or accurate.

  • More source observations did not mean more validated misinformation.

    Claude returned 455 observations and Gemini returned 140. The inventories also contained corrective, authoritative or otherwise contextual material.

  • An exact URL match did not settle eligibility.

    Downstream assessment retained 86 Claude and 21 Gemini page targets for manual review even though those aggregated pages also had exact URL matches within the saved source evidence.

  • The later assessment was not independent across providers.

    Both discovery inventories passed through the same Gemini 3.7 Flash-based structured assessment process and policy snapshot.

  • The evidence is descriptive.

    Each provider contributed one completed full run. The archive does not establish advance registration of the design, repeatability, audience reach or any measured intervention effect.

Evidence status: Kojable's study figures and processing details are author-reproduced from its September 2026 saved records and accompanying code. They have not been independently audited for this article. The limitations and source notes below identify the remaining qualifications.

Source-discovery process

  1. Input

    P01–P10, one completed Claude and one completed Gemini source-discovery run, 11 September 2026 UTC.

  2. Mechanism

    preferred-URL exclusion, conservative normalisation and deduplication; common Gemini Step 2 assessment and policy snapshot.

  3. Bounded output

    412 Claude and 108 Gemini URL identities, 21 shared, and distinct retained/manual-review/reference-only aggregate dispositions.

1. Research question and public-health context

The question was technical: what can AI-assisted research reliably establish about the sources it finds when investigating vaccine misinformation, and where must human verification begin?

An online claim may be quoted critically, discussed in neutral reporting, repeated without context, or used to promote a misleading interpretation. A research process that finds a relevant page has not necessarily established which of these is happening. The same underlying narrative may also travel between articles, videos, podcasts, transcripts and reposts without creating independent evidence each time.

Childhood vaccine–autism narratives offered a sensitive test setting because scientific interpretation and communication decisions carry real consequences. The scientific baseline is not an unresolved contest between opposing claims. On 27 November 2025, the World Health Organization's Global Advisory Committee on Vaccine Safety (GACVS) assessed updated systematic reviews and reaffirmed that the high-quality evidence does not support a causal relationship between vaccines and autism spectrum disorder. WHO published its committee statement on 11 December 2025. A separate WHO evidence review published on 3 September 2026 also considered evidence on vaccines, thiomersal and autism.

The monitoring task was therefore not to reopen an established scientific conclusion. It was to identify specific pages and documentary contexts for closer examination while distinguishing potentially misleading material from legitimate medical information, neutral discussion and correction.

The ten archived monitoring tasks, P01–P10, covered the following themes. These are concise descriptions of their subject matter, not a claim that the study was preregistered:

Monitoring tasks P01–P10 and their vaccine-misinformation research themes.
Task Monitoring theme
P01 Direct vaccine–autism causation claims
P02 Thiomersal, thimerosal and mercury narratives
P03 Aluminium-adjuvant interpretations
P04 Childhood vaccination schedules and immune-overload narratives
P05 Developmental-regression accounts presented as causal evidence
P06 Interpretation of safety reports, surveys and observational data
P07 Reuse of retracted or disputed research
P08 Legal, compensation, whistleblower and institutional narratives
P09 Podcasts, videos and cross-platform republication
P10 Commercial products, treatments and alternative-schedule claims

This was targeted discovery of pages relevant to particular research themes. It was not a prevalence survey of online vaccine information, a measure of population beliefs or a test of what an ordinary user would hear from a consumer AI assistant.

2. Study design and archived provenance

The comparison uses one completed full discovery run for each provider, both collected on 11 September 2026 (UTC). Each completed the same ten tasks.

Completed full source-discovery runs on 11 September 2026 (UTC).
Step 1 provider Recorded model ID Collection window (UTC) Completed tasks
Claude claude-sonnet-5 14:47–15:00 P01–P10 (10/10)
Gemini gemini-3.7-flash 14:24–14:29 P01–P10 (10/10)

Two Gemini single-task attempts were excluded from the comparison. They were not treated as additional completed runs or repetitions. The archived material establishes the two complete runs and their task coverage; it does not establish that the research design was specified prospectively or that another run would reproduce the same source lists.

One discovery stage, one common assessment process

The providers differed at Step 1 source discovery. Subsequently, both inventories underwent a common Step 2 structured assessment using gemini-3.7-flash with medium thinking, an enforced output structure and the same policy snapshot. The pipeline then produced source ledgers, URL matches and run-level aggregates.

That common processing stage matters. The Step 2 categories are not two independent medical or editorial judgments produced by Claude and Gemini. They are classifications produced under a shared downstream process, with its own limitations.

The retained research evidence includes run manifests and raw responses, normalised Step 1 source records, the URL-normalisation and verification rules, Step 2 source ledgers, run aggregates and 51 batch audits. The reported counts were reconciled by task and source identifier across these records. The original exported intersection list was not located, but the stated URL-handling rules reproduce the 21 shared identities and 499-identity union. A metadata mismatch between the Step 2 schema named in the manifests and in the batch audits remains documented in the limitations.

3. What the source-discovery runs returned

The first layer is the discovery inventory: source observations recorded in Step 1, including repeated appearances across the ten tasks. An observation is not automatically a unique page, a misinformation incident or a final-answer citation.

Source observations and usable preferred-URL identities by provider.
Step 1 measure Claude Gemini
Source observations across P01–P10 455 140
Observations with missing or unusable preferred_url 1 16
Observations with usable preferred_url 454 124
Distinct normalised preferred-URL identities within the run 412 108

Source: Kojable's archived Step 1 records and author-reproduced URL reconstruction for the 11 September 2026 full runs. The exclusion rule applies to the preferred-URL comparison, not to the source-observation totals.

Claude produced more recorded source observations in this pair of runs. The comparison does not isolate why. Differences could involve the providers' research behaviour, the sources exposed by their search tools, URL resolution or the information available to the extraction process. The saved output volumes cannot, by themselves, distinguish these explanations.

More observations can also mean more duplicated or unresolved material requiring assessment. The operational question is not simply how many sources appeared, but how many can be identified, evaluated and handled according to the purpose and safety requirements of the research.

4. Run-wide preferred-URL overlap was 4.21%

To compare like-for-like discovery identities, the reconstruction used each source observation's saved preferred_url. Missing or unusable preferred URLs were excluded from this calculation: one Claude observation and sixteen Gemini observations.

The repository normaliser removed www., URL fragments and known tracking parameters, while preserving the URL scheme and query parameters that identify a page. The reconstruction then deduplicated identities across all ten tasks within each provider's full run. No additional alias merges were applied.

The resulting run-wide sets were:

Conservative preferred-URL overlap across ten tasks.
URL-set measure Count
Distinct Claude URL identities 412
Distinct Gemini URL identities 108
Shared identities (intersection) 21
Identities in either set (union) 499

The Jaccard index compares shared identities with all identities appearing in at least one set:

Jaccard overlap = shared URL identities ÷ all distinct URL identities = 21 ÷ (412 + 108 − 21) = 21 ÷ 499 ≈ 0.0421, or 4.21%.

That is 4.21% overlap of the two recorded, normalised preferred-URL discovery sets. The numerator and denominator include sources across the Step 1 inventories, regardless of the classifications they later received. They do not describe a set restricted to misinformation targets or eligible outreach candidates.

What low discovery overlap means—and does not mean

Under these collection conditions, Claude and Gemini exposed largely different URL identities for the same ten monitoring tasks. A URL found by both may be a useful review-priority signal; however, shared discovery is not proof that a page is credible, misleading, influential or widely read. A URL found by only one provider may also be important.

The result depends on the identity rule. Redirects, syndicated copies and apparent aliases may remain separate where the archived evidence does not justify merging them. A more aggressive deduplication procedure could produce a different number, but should not be silently substituted for the documented calculation.

Most importantly, discovered source overlap is not final-answer citation overlap. The study did not collect a matched panel of ordinary user-facing medical answers and compare their displayed citations. Nor does one run per provider establish a stable characteristic of future model behaviour.

5. A matched page was not necessarily an eligible target

The later Step 2 assessment produced a different kind of record: run-level classifications and review dispositions. These aggregate figures should not be added to the Step 1 observation counts as if they were successive, mutually exclusive buckets of the same denominator.

Step 2 run-aggregate dispositions by provider.
Step 2 aggregate category Claude Gemini Interpretation
Retained page targets with exact-page match 38 17 Page-target records retained by the pipeline under its assessment rules
Page targets held for manual review 86 21 Review required; withheld from automatic targeting despite exact URL matching
Reference-only records 276 65 Kept for context, correction or calibration; not targeting-eligible

Source: Kojable's author-confirmed Step 2 run aggregates. These are downstream aggregate measures, not unique preferred-URL set sizes or independently confirmed misinformation events.

Exact-page identity: a narrower claim than medical verification

The archived label exact-page verified indicates that a normalised candidate URL matched a URL attached to a linked Step 1 source record. For Claude, retained pages matched observed URLs; for Gemini, they matched resolved URLs. This establishes a page-identity relationship within the saved evidence.

It does not establish that the live page was independently revisited, that its content remained unchanged or that a medical expert verified the factual claims attributed to it.

The manual-review queue makes the distinction particularly clear: all aggregated manual-review pages also had exact URL matches. Identity matching was therefore not sufficient to confer automatic targeting eligibility.

These decisions are better represented on two axes, consistent with Kojable's wider measurement terminology:

Separate evidence-verification and action-eligibility questions.
Assessment axis Question Example in this study
Evidence verification What identity or claim was actually checked, and to what standard? A candidate URL matched a linked Step 1 source URL; the underlying claim could still require review
Action eligibility Is it appropriate to proceed under the applicable research and safety rules? A page was retained, held for manual review or classified as reference-only

For public-health research, a further distinction is indispensable: what the page actually says, whether it asserts or criticises a claim, and how the relevant scientific evidence should be interpreted are not answered by an exact URL match.

The 86 and 21 manual-review records are therefore not 107 independently confirmed misinformation incidents. They are review obligations created by the study's classification rules.

Why reference-only sources matter

Authoritative medical guidance, research papers, fact-checks, neutral reporting and other contextual sources may surface in a targeted source-discovery process. Some are needed to evaluate claims; others may accurately explain why a claim lacks support. Their presence does not make them misinformation publishers or suitable targets for a corrective response.

The reference-only classification helps preserve that distinction. It is a role assigned within the research process, not an independent guarantee of the quality of every record in the category. For any individual source, reviewers still need to examine the document and its relevance to the question at hand.

6. How this case relates to Kojable's source-research measurement framework

Kojable's AI source research measurement framework proposes separate definitions for discovery, source identity, observed citations, shortlisting, verification, content gaps and interventions. The September runs preceded publication of that framework. The mapping below is retrospective: it shows how this case can be described using the framework's vocabulary, not that the framework controlled the original research or has been validated by it.

Retrospective mapping to seven source-research measurement objects.
Framework research object What this case supports What remains outside its evidence
Discovered source 455 and 140 Step 1 source observations Exhaustive coverage of the web
Unique source 412 and 108 normalised URL identities, with 21 shared across a union of 499 Alias-complete document identity or distinct publisher counts
Observed AI citation Not measured Any claim about final-answer citation frequency
Shortlisted opportunity A context-specific analogue: retained monitoring targets and manual-review queues Commercial outreach suitability or confirmed interventions
Verified evidence Recorded URL-identity matching Comprehensive fact-checking of each page or clinical review
Content gap Possible information needs suggested by the monitoring themes Systematically coded or exhaustively established information gaps
Proposed intervention Educational and corrective response ideas Publication, uptake, audience effect or causal improvement

The framework distinguishes an observed-answer track, where a captured AI answer and its exposed citations are measured, from an independent-research track, where relevant sources can be discovered whether or not they appear in an AI answer. This case belongs to the latter discovery-oriented analysis; its Step 1 records must not be relabelled as observed AI citations.

The framework also addresses commercial publisher and creator qualification. The underlying measurement discipline may transfer across contexts, but public-health monitoring has different safety, ethical and professional requirements. A page deserving assessment is not automatically an outreach opportunity.

This research relationship complements, without pooling results from, Kojable's separate source-identity study and cross-provider source-pool research.

7. Interpretation risks and responsible responses

The ten monitoring themes suggest several analytical categories of interpretation risk. These are ways to organise a subsequent human review, not four independently validated or prevalence-weighted clusters produced by the experiment.

Interpretation risks requiring specialist review.
Interpretation challenge What requires review
Causal interpretation Whether chronology, personal experience or observed association is being presented as proof of causation
Scientific evidence Whether ingredient descriptions, study designs, comparison groups, uncertainty and strength of evidence are represented fairly
Institutional and documentary records What legal, compensation, retraction or institutional documents actually establish, without turning administrative outcomes into scientific proof
Amplification and commercial pathways Whether repeated media appearances represent independent evidence, and whether claims are connected to products, services or potentially unsafe advice

A responsible researcher would preserve the relevant page, examine it in full context, distinguish quotation from endorsement and check the underlying evidence. Where medical or legal interpretation matters, the assessment requires appropriately qualified expertise.

Proposed content responses are not measured interventions

The accompanying editorial analysis suggested possible educational responses to information needs revealed by the monitoring themes. They were proposals, not a deployed campaign or an evaluation of public understanding.

Proposed educational response types and safeguards.
Proposed response type Potential use Essential safeguard
Evidence explainers Clarify causation, study design and scientific terminology Clinical or epidemiological review
Data-literacy material Explain denominators, comparison groups, uncertainty and safety-report interpretation Biostatistical or pharmacovigilance review
Document and claim chronologies Show what primary scientific, institutional or legal records establish Primary-record checks and specialist legal review if relevant
Consumer-safety and communication resources Address misleading treatment promises, sensitive narratives or recurring public questions Clinical review, proportionality and non-amplification safeguards

Not every discovered page calls for a response. Some warrant archiving or monitoring; others may call for a clarification, specialist evidence packet or referral to established authoritative information.

An author-proposed monitoring sequence

The following is a practical synthesis from this case, not a WHO-prescribed seven-step protocol:

  1. Capture

    the page, collection time and relevant surrounding context.

  2. Resolve

    the source and page identity, preserving unresolved cases.

  3. Classify

    whether the material promotes, amplifies, neutrally discusses or corrects the claim.

  4. Verify

    the underlying documentary or scientific evidence using the appropriate specialist standard.

  5. Assess risk

    and determine whether public engagement could create avoidable amplification.

  6. Prepare

    a proportionate response, if justified, for qualified human review.

  7. Evaluate

    any action actually taken, keeping observed outcomes separate from the initial discovery signal.

WHO's Social listening in infodemic management for public health emergencies: guidance on ethical considerations, published 8 April 2025, provides a relevant ethical reference for listening, interpreting community information needs and considering sensitive, proportionate responses in the context of public-health emergencies. The ECDC's systematic scoping review of social-media monitoring and vaccine hesitancy supplies related research context. Neither source is represented here as validating this article's specific seven-step sequence or its AI pipeline.

8. Limitations and unresolved evidence

These results are descriptive and exploratory. The most important boundaries are part of the finding, not a footnote to it.

  • One completed run per provider.

    The two full runs covered the same ten tasks, but there were no controlled repetitions. Two Gemini single-task attempts were excluded. The archive does not establish preregistration or stable provider-level performance differences.

  • No causal explanation for the inventory difference.

    The recorded output does not isolate the contributions of provider behaviour, search exposure, retrieval conditions, extraction or URL resolution.

  • No prevalence or audience-impact estimate.

    Tasks deliberately sought particular narratives. Source observations cannot quantify the share of online material that is misleading, how widely material was encountered, how persuasive it was or what effect it had.

  • No clinical model ranking.

    The study did not evaluate the medical accuracy or safety of Claude's and Gemini's consumer-facing advice. WHO's scientific evidence is external context, not a validation label generated by either model.

  • No final-answer citation benchmark.

    A source recorded in the discovery inventory is not evidence that an AI assistant cited it in a final answer.

  • No comprehensive claim validation.

    Exact-page matching establishes a recorded URL relationship, not a medical fact-check. The study preserved manual-review and reference-only distinctions for that reason.

  • Shared downstream assessment.

    Both inventories were classified using the same Gemini-based Step 2 process. Its outputs should not be described as independent confirmation by two assessment models.

  • A schema-metadata discrepancy remains.

    The 51 saved batch audits identify step2.batch.v3 as the expected and parsed schema, while the associated run manifests list step2.batch.v2. This inconsistency is recorded; the current evidence does not establish that it altered the aggregate counts or classifications.

  • Identity resolution is intentionally conservative.

    One Claude and sixteen Gemini preferred URLs were unusable for overlap analysis. URL variants, redirects or aliases not justified by the saved identity rule may remain distinct. The original intersection export was not found, although the author-reported reconstruction reproduces the stated figures.

  • No measured response effect.

    Educational materials, review priorities and communication responses were proposed, not established as completed or effective interventions in this study.

  • No independent archive audit for this publication.

    The metric reconciliation, model metadata and classification details are reported from Kojable's retained first-party records. The underlying Windows-local archive has not been independently inspected by the editorial workflow preparing this article.

The results should not be extended beyond the date, tasks, models, preferred-URL rules and recorded assessment process without additional data. Repeated runs, a fixed comparison protocol, deeper page resolution and qualified human assessment would be needed to evaluate stability and classification quality.

9. What the research contributes

The strongest finding is a measurement distinction, not a model ranking.

In this pair of source-discovery runs, the providers returned largely different preferred-URL sets. That difference makes a useful inventory, but it does not make every discovered source a unique document, every identified document a misinformation case, or every source overlap an AI citation.

The manual-review and reference-only records add a second lesson. Page identity, claim interpretation and eligibility for action are separate decisions. The separation is especially important when a source concerns public health and a careless response could amplify misleading material or mischaracterise a corrective source.

For Kojable's wider AI answer alignment research, this case illustrates See in the See → Align → Prove → Compound process by documenting what was observed and where the evidence ends. Align, Prove and Compound describe later work: justify a change, test what changes under comparable conditions, and use the findings to guide the next decision. The case establishes no guarantee of citations, public-health improvements or control over AI answers.

A stronger follow-up study could preregister matched tasks, repeat both discovery runs under recorded conditions, audit candidate pages independently, resolve document aliases and test reviewer agreement on claim and eligibility classifications. Any later information intervention would need its own ethically appropriate outcome design.

The responsible output of AI-assisted misinformation monitoring is not the largest possible list of suspicious pages. It is a traceable account of which sources were discovered, what they can support, which judgments remain open and what response—if any—is justified.

Frequently asked questions

Did the study show that Claude or Gemini spreads more vaccine misinformation?

No. It compared targeted source-discovery inventories from one completed run per provider. It did not measure the content or medical accuracy of ordinary consumer AI answers, or how often a model might repeat a misleading claim.

Why did Claude return 455 source observations and Gemini 140?

Those are the recorded Step 1 totals. The study did not experimentally separate provider search behaviour, source exposure, URL resolution or extraction effects, so it cannot establish why the counts differed or which inventory was more complete.

How was the 4.21% overlap calculated?

After excluding unusable preferred URLs, conservatively normalising and deduplicating across the ten tasks, Claude had 412 distinct URL identities and Gemini had 108. The sets shared 21 identities, leaving 499 in their union. Jaccard overlap is 21 / 499, or about 4.21%. No alias merges were added.

Does a page appearing in both inventories make it more important?

Not necessarily. Shared discovery may help prioritise a page for examination, but does not establish accuracy, misinformation status, audience reach, influence or response eligibility. A page found by one provider may still be significant.

What did “exact-page verified” mean?

It meant that a normalised candidate URL matched a linked source URL within the saved Step 1 evidence. The status concerns page identity. It does not certify the truth of the page's claims or replace independent medical review.

Why were pages still held for manual review when their URLs matched?

Because identity is only one check. The pipeline retained 86 Claude and 21 Gemini page targets for manual review despite exact URL matches. Further assessment of source stance, evidence, context and action eligibility can remain necessary.

What are reference-only records?

They are sources retained for context, correction or calibration rather than automatic targeting. The category may include medical authorities, refutations and neutral reporting; it is a research disposition, not an endorsement of every individual source.

Did the study test whether its proposed content responses worked?

No. The educational response types and monitoring sequence were proposals. The archive did not establish a completed communication campaign, a change in public understanding or a causal intervention effect.

How is this case study related to Kojable's measurement framework?

The measurement framework defines distinct research objects and their reporting requirements. This case retrospectively illustrates several of those distinctions with source-discovery and verification data. It does not validate the framework's effectiveness or convert public-health monitoring candidates into commercial opportunities.

Sources and methodological references

First-party research records

  • Kojable, archived Vaccine-Autism Misinformation Monitor, Claude full run collected 11 September 2026, Step 1 model claude-sonnet-5; run manifest, raw and normalised Step 1 records, Step 2 source ledgers, audits and run aggregate.
  • Kojable, archived Vaccine-Autism Misinformation Monitor, Gemini full run collected 11 September 2026, Step 1 model gemini-3.7-flash; corresponding manifest, records, source ledgers, audits and aggregate.
  • Kojable repository rules for preferred-URL identity, normalisation and exact-page matching; author-reproduced overlap reconstruction across P01–P10. The underlying research archive is not linked as a public download here.

External scientific and ethical references

About Kojable

KojableAI answer alignment platform

Kojable is an AI answer alignment platform for B2B companies. Its See → Align → Prove → Compound process connects observation and diagnosis with evidence-backed changes, comparable testing and the next decision. This case study is descriptive first-party source research, not clinical advice.