How Kojable Made DataForSEO LLM Mentions Production-Safe

LLM Mentions is not equivalent to Kojable sending a fresh prompt to ChatGPT or Google whenever collection runs. DataForSEO returns recorded AI-search mention observations. Provider rows include first-observed and last-updated timestamps, and both may predate the moment Kojable retrieved them. A synchronous, or “live”, API response describes the retrieval path; it does not necessarily mean a new underlying AI answer was generated at that instant.

Market and platform scope matters just as much. The provider’s google platform represents Google AI Overview; we do not describe it as Google AI Mode. ChatGPT LLM Mentions coverage is market-specific. Our first implementation planned Google and ChatGPT work for every workspace without first establishing the exact capability combination.

That was our mistake. In the Irish production example, the workspace had 14 tracked topics. Planning two platforms produced 28 originally planned observations, but only 14 Google AI Overview observations were supported. The 14 ChatGPT observations were unsupported for that market and should never have reached paid planning. The correction is a capability check, not a criticism of a provider whose market availability needs to be interpreted as documented.

The request is driven by tracked topics, not the brand domain alone. Broad topics can return answers whose words overlap the query while their practical meaning is only loosely related to the workspace. Treating every returned row as relevant would give those tangential observations the same metric weight as a direct buyer question.

Kojable therefore uses public-level signals such as topic overlap, question meaning, answer meaning, workspace context, brand signals and domain signals. The classifier is deterministic and versioned so the same policy can be audited and compared, but it remains heuristic. It can falsely exclude relevant observations, falsely accept weak ones and bias against broad or short topics. Ambiguous rows need future review, while all policy versions need periodic calibration.

The 309 excluded rows were not failed, corrupted or necessarily invalid. They were provider rows that did not satisfy Kojable’s relevance policy for the core metrics. Preserving that distinction is essential to treating the external dataset fairly and keeping metric meaning honest.

Portfolio deduplication

The same provider response may appear under more than one tracked topic. Topic views retain that topic-local relationship because it explains where the response was found. Portfolio-level Share of Voice counts the underlying response once, preventing a repeated observation from gaining extra influence merely because several topics retrieved it.

Why search results are not automatically citations

Provider sources represent pages cited or relied on in the final answer. Provider search_results represent broader retrieval output. A retrieved page may have helped exploration without influencing the final answer, so counting every search result as a citation would inflate citation metrics. Kojable retains retrieval evidence separately and creates the 2,938 citation events from source records.

Provider ingestion has to tolerate ordinary variation without hiding genuine infrastructure failure. The revised boundary validates typed provider envelopes, handles null values safely, records unknown platforms, detects missing task results and bounds large Markdown answers. Bounded transport retries apply only where a retry is safe; authentication problems fail fast. Telemetry is redacted before it becomes operational evidence.

Long answers exposed a less obvious text problem. JavaScript strings use UTF-16, so a naive slice can split a surrogate pair and damage a character such as an emoji. Safe truncation respects that boundary. Kojable stores a bounded answer excerpt, a content hash and the original length, which preserves useful evidence without persisting an unbounded response.

That item-level quarantine is not a licence to swallow unexpected infrastructure failures. Observation failures are isolated by topic and platform, while failures that threaten the integrity of the broader system remain visible. The parent collection can finish as partial when valid data exists alongside degraded siblings.

The schema mismatch was ours

DataForSEO returned model_name, while an initial Kojable PostgreSQL schema used model. That was a Kojable integration contract failure, not provider-data scarcity. The corrected system reconciles the contract safely and treats schema readiness as a deployment requirement.

Recovering the historical run

The production run persisted valuable data but ended with a failed parent-run status. The read model behaved correctly by ignoring failed runs, yet the consequence was that usable evidence disappeared from the product. We repaired the historical status using only persisted PostgreSQL evidence. No additional provider call was made, no missing rows were reconstructed, and the original failure remained auditable. Future degraded runs can surface as usable partial snapshots.

Monitor: collect eligible topic, answer, source, platform and provider-time evidence. The goal is not a score in isolation; it is a traceable observation with a declared scope.

Diagnose: separate unsupported markets, sparse samples, irrelevant rows, citation evidence, ownership gaps and collection failures. A useful diagnosis begins by refusing to call every absence the same thing.

Improve: change the information environment, topic scope, evidence or product configuration based on that defensible diagnosis. Kojable does not control third-party AI systems, so actions focus on evidence and configuration that teams can actually influence.

Verify: run comparable future collections and say whether the result is exact, sampled or directional. The AI visibility monitoring approach is useful only when the comparison conditions are visible.