# Can you trust the numbers an AI visibility platform reports?

> The short answer is: AI visibility platforms can produce useful directional data, but their numbers are not automatically comparable, reproducible or representative of every user’s experience.

Source: https://bestaivisibilityplatforms.com/ai-visibility-data-accuracy · Publisher: Platform Ledger · Updated: 2026-11-15

## TL;DR
- AI visibility scores are best treated as directional signals unless the vendor discloses sampling, reproducibility, platform coverage and scoring logic to a decision-grade standard.[1]
- Browser and API collection measure different configurations, so buyers should require engine-by-engine disclosure rather than assuming that an API answer represents the consumer application.[2]
- Repeated prompts can produce different brands, citations and rankings, making a single screenshot an inadequate test of AI visibility data reliability.[3][9]
- Location, memory, history, language and account state can affect AI answers independently, so geography and personalisation must be tested as separate variables.[22]
- The strongest pilot combines repeated raw-answer sampling with manual mention audits, transparent denominators and documented model, prompt and collection-method changes.[3]

## Why AI visibility data is difficult to trust

The short answer is: **AI visibility platforms can produce useful directional data, but their numbers are not automatically comparable, reproducible or representative of every user’s experience**. Trust depends on four things: what surface the vendor measured, how it sampled prompts and locations, how it detected mentions and citations, and how transparently it converted raw answers into scores.[1][2]

The IAB’s 2026 framework distinguishes **directional measurement** from **decision-grade measurement**. Directional data can identify patterns and trends, while decision-grade data requires stronger evidence across query volume, sample size, prompt coverage, testing cadence, reproducibility and platform coverage.[3]

A score can therefore be accurate about the answers a vendor collected while saying little about the answer a particular customer sees. Before comparing tools, separate these questions:

1. Did the platform collect the intended AI product experience?
2. Was the prompt set relevant to the buyer’s market and audience?
3. Did it correctly identify mentions, citations and competitors?
4. Can the reported score be reconciled to the underlying answers?
5. Is the observed change larger than normal run-to-run variation?

## What does an AI visibility score actually measure?

There is no single universal definition of “visibility.” A platform may measure whether a brand appears in an answer, whether its domain is cited, where it is listed, how often it appears relative to competitors, or an estimated exposure value.[4]

| Measure | How public documentation defines it | Buyer implication |
|---|---|---|
| Brand mention | Ahrefs counts three occurrences of a brand in one answer as one mention.[4][2] | Confirm whether the unit is an answer, an occurrence or another event. |
| Visibility rate | Peec divides responses mentioning a brand by total responses.[5][4] | The denominator is essential when comparing vendors. |
| Visibility rate | Profound defines its comparable metric using responses mentioning at least one brand.[4][6] | Two similarly named percentages may use different denominators. |
| Share of voice | Peec describes this as a brand’s mentions relative to the tracked competitive set.[4][7] | Keep the competitor set fixed between periods. |
| Estimated impressions | Ahrefs weights mentions using Google search volume and says the relationship to AI-tool query frequency is not validated.[1] | Treat the result as modeled exposure, not measured AI demand. |
| Citation | Ahrefs separates visibly cited pages from pages found during background retrieval.[8] | Do not count every retrieved page as a visible citation. |

The important question is not which tool reports the highest percentage. It is whether the numerator, denominator, weighting and reporting window match the business decision.

Ahrefs’ Brand Radar methodology illustrates the boundary. Its questions are built from Google’s People Also Ask corpus and Ahrefs’ keyword database, so its coverage follows available keyword data rather than measured AI usage in every country.[1] Its estimated-impression metric is consequently a model of potential exposure, not a count of users who saw or acted on an AI answer.[9][1]

A buyer should keep at least four measures separate:

- Brand mentions.
- Visible citations.
- Modeled exposure.
- Website visits, leads or revenue attributed to AI.

Combining them into one headline score can make a dashboard easier to read, but it can also conceal what the number actually represents.

## How do AI visibility tools collect answers?

The collection method affects what a platform observes. Browser or front-end collection is intended to reproduce a public consumer experience, while API collection offers explicit configuration but may not match the consumer application. Some vendors document a mixed approach.[10][2]

| Collection method | What it captures | Known gaps | Which vendors say they use it |
|---|---|---|---|
| Browser or front-end collection | Answers returned by a public AI website or search interface.[11] | Account state, personalization, geography and session-reset procedures may not be fully disclosed.[12][13] | Profound, Peec AI and Ahrefs publicly describe front-end or public-interface collection.[13] |
| Programmatic submission through AI websites | Automated prompts submitted to AI websites rather than manually operated sessions.[9][13] | The vendor may not expose the exact browser state, account tier or retry policy. | Otterly.AI documents this approach.[14][12] |
| Official API collection | Responses generated through an API with explicit model, tool and location settings.[7][15] | The API product may not reproduce the consumer application’s interface, memory, search behavior or ranking logic.[2] | Otterly.AI documents an API exception for Claude.[16] |
| Mixed browser and API collection | Different methods selected for different platforms.[16][2] | Buyers need an engine-by-engine mapping to know which method applies.[2] | Scrunch documents both browser automation and official APIs.[16] |
| Undisclosed upstream transport | A vendor reports tracked answers without publicly specifying whether the upstream collection used a browser, API or both.[4] | Surface equivalence, account context and reproducibility cannot be independently assessed from the reviewed documentation.[2] | Finseo’s reviewed public documentation does not specify the upstream transport.[11][17] |

The distinction is not “browser good, API bad.” APIs can be valid measurement surfaces when the intended configuration is documented. The problem occurs when an API result is presented as evidence of what every consumer sees, or when a browser result is treated as free of personalization and location effects.

The IAB recommends that providers disclose platform coverage, model versions, prompt construction, collection architecture, attribution logic, factual-accuracy classification and baseline changes.[2] A serious buyer should request those details for every engine rather than accepting one platform-wide label such as “real-time” or “real user data.”

## Why can API answers differ from consumer AI answers?

ChatGPT’s consumer search experience can automatically decide when to search, rewrite a question into targeted queries, use relevant saved memories and use approximate IP-based location.[18] OpenAI’s API exposes a separately configured web-search tool with controls for search invocation, location and live or cached access.[10]

Google’s Gemini API similarly allows the `google_search` tool and lets the model decide whether search would improve an answer; it may issue one or more searches.[9][10] Google also distinguishes AI Overviews and AI Mode, which can use different models and techniques and therefore produce different responses and links.[19]

| Product surface | Documented behavior | What a buyer should record |
|---|---|---|
| ChatGPT consumer search | Automatic search, query rewriting, memory and approximate location can affect the experience.[18] | Search mode, account state, memory/history controls and location. |
| OpenAI API | Web search is explicitly configured and can include location and live-access settings.[10] | Model identifier, search-tool settings and evidence that search actually ran. |
| Gemini API | Grounding with Google Search can be enabled, but the model decides whether to use it.[20][11] | Whether grounding was enabled and whether searches were issued. |
| Gemini Apps | Personalization may use previous chats and user instructions.[21] | Whether personal context was present. |
| Google AI Overviews | A Search surface with its own behavior and coverage.[1] | The exact Search surface, country, language and timestamp. |
| Google AI Mode | A separate Search surface that can use continuity from prior searches.[22] | AI Mode should not be silently treated as equivalent to AI Overviews. |

Model labels also do not guarantee an unchanged system. OpenAI documents snapshots as a way to lock a particular model version, while aliases such as `chat-latest` can point to an updated underlying model.[23]

A vendor does not need to claim that its API and browser results are identical. It does need to explain which one it measures and why that surface answers the buyer’s question.

## How much do repeated prompts vary?

Repeated prompts can produce different brands, citations, ordering and framing even when the text is unchanged. SparkToro and Gumshoe studied 600 volunteers, 12 prompts and 2,961 responses across ChatGPT, Claude and Google’s AI search experiences; participants used normal settings, so country, device, history and temperature were not controlled.[9]

The study reported less than a 1% chance of repeating an unordered recommendation list and less than 0.1% for repeating the same ordered list.[9][3] That does **not** mean a visibility platform is wrong 99% of the time. It means that exact recommendation lists can be highly unstable.

The same research found recurring brand inclusion. In 994 responses to 142 headphone prompts, Bose, Sony, Sennheiser and Apple appeared in 55–77% of responses.[9] A complete list can therefore change while individual brands remain comparatively common.

A separate 2026 API study tested 4,500 calls across three providers, with five runs per test cell. All five runs produced the same brand set in 15.0% of cells, mean Jaccard overlap was 0.54, and the first-named brand remained the same in 75.1% of runs.[24] These results are not interchangeable with the SparkToro findings: they used different surfaces, prompts, sample sizes and experimental controls.[9][2] Citations show similar churn: [Finseo's own study of about 1.4 million German-language (DACH) AI answers](https://www.finseo.ai/blog/ai-citation-stability) (Finseo is built by the team behind this site) found that only 38.4% of URLs cited by ChatGPT were still cited the next day for the same prompt, compared with 65.5% on Perplexity.

A study of hosted models also reported accuracy differences of up to 15% across equivalent runs, even under settings including temperature zero, top-p one and a fixed seed.[15] Its task was model accuracy rather than brand detection, so it should not be treated as an AI visibility error rate.[2][15]

> A single captured answer is a sample, not a census of what an AI platform will say.

### Why can repeated prompts return different brands?


Why can repeated prompts return different brands?

AI systems may retrieve different sources, use different search queries, select different passages or generate a different ordering of otherwise relevant brands. Consumer products can also incorporate location, account history, memory, device context and changing model or search configurations.[18][12]

The practical consequence is that a vendor should measure repeated inclusion, citation stability and position stability separately. One matching screenshot proves little, and one mismatch does not prove that the platform’s entire dataset is unreliable.[2]



## How do location and personalisation affect reliability?

Location and personalisation are related but distinct. OpenAI documents approximate IP-derived location and saved memories as inputs that may affect ChatGPT search-query construction.[18] Perplexity documents profile preferences, inferred network location, memory and previous searches; it says memory and search history are disabled in incognito mode.[22][14]

Google says that location can continue to affect Search results even when personalisation is disabled.[12][25] Google AI Mode separately documents continuity from previous searches and personalisation controls.[22]

| Factor | Why it matters | What to control in a pilot |
|---|---|---|
| Country | Search results, available products and cited sources can differ by market.[9][10] | Country, network location and market setting. |
| Language | Prompt language can change retrieval and answer composition.[26] | Exact prompt language and translation policy. |
| Account state | Memory, history and profile instructions can influence answers.[21] | Logged-in or logged-out status, account tier and fresh-session policy. |
| Device and network | Search systems can use device and approximate location signals.[18][10] | Device type, IP region and any proxy configuration. |
| Timestamp | Models, indexes and search results change over time.[9][4] | Collection timestamp and model version. |
| Personalisation | A non-personalised setting does not necessarily eliminate geographic effects. | Test geography and personalisation as separate variables. |

“Non-personalised” is therefore not the same as “location-neutral.” Conversely, matching a country does not prove that two measurements used the same account history, memory or search context.

## Can platforms correctly detect brand mentions and citations?

Brand detection is its own accuracy problem. Systems must decide whether a name is a brand, whether a spelling variation refers to that brand, whether a parent and sub-brand should be combined, and whether an indirect reference counts.[2]

Academic brand-linking research found that high precision can coexist with substantially lower recall. One reported comparison produced 97.22% precision and 64.66% recall for a named-entity-recognition plus exact lexical-match system.[2] Those are results from product-search queries, not from any AI visibility vendor’s generated-answer dataset.[7][2]

Ahrefs describes mention detection as string matching, while academic entity-linking systems attempt to resolve names to catalog identities.[2] Neither description provides a vendor-specific false-positive or false-negative rate.

During a pilot, ask the vendor to distinguish:

- Exact brand mentions.
- Aliases and spelling variants.
- Abbreviations.
- Parent and sub-brand relationships.
- Competitor names embedded in URLs or citations.
- Negative or hypothetical mentions.
- Mentions in languages other than English.
- Implicit references that do not use the brand name.

The right validation metrics are **precision and recall**, not one unexplained “accuracy” percentage. Independent brand-detection accuracy for the vendors reviewed in the report is not published.[9][15]

## What do vendors disclose about cadence and sampling?

Collection cadence is not the same as data freshness or sample size. Profound, Peec and Otterly describe daily prompt collection, while Scrunch documents daily collection for an initial period followed by collection every 72 hours.[27]

Ahrefs describes monthly chatbot refreshes with a 90-day reporting window, while its Google AI datasets use different refresh behavior.[1] Semrush documents separate cadences for AI Analysis, Prompt Tracking and Brand Performance.[7]

| Vendor | Publicly documented cadence | Repeated samples per prompt |
|---|---|---|
| Profound | Daily tracked prompts.[13] | Not published. |
| Peec AI | Daily monitoring, with accepted prompts entering a 24-hour cycle.[28] | Not published.[11][4] |
| Otterly.AI | Daily checks without a fixed hour.[29] | Not published. |
| Semrush | Product-specific daily and weekly update schedules.[30][7] | Not published. |
| Ahrefs Brand Radar | Monthly chatbot refreshes and a 90-day reporting window; Google datasets use separate refresh cycles.[1] | Not published.[1] |
| Scrunch | Daily during the first 14 days, then every 72 hours, with manual refresh available.[27] | Not published. |

A dashboard that says “this month” may contain answers collected at different times, under different model versions and with different retry or failure policies. Buyers should request attempted runs, successful runs, retries, missing-response handling and raw timestamps.

## How should you test an AI visibility vendor during a pilot?

The IAB recommends variability baselines, geographic and temporal controls, platform-change tracking and baseline management when models change.[26] Use those principles to test evidence rather than screenshots.

1. **Define the product surface.** Record the exact engine, consumer application or API, model identifier, search configuration and whether search actually ran.[2]
2. **Freeze the prompt set.** Use a documented set of commercial, informational, comparison and branded prompts, with fixed language and competitor definitions.[2][7]
3. **Control the context.** Specify country, language, device, IP region, logged-in state, account tier, memory, history and profile settings.[18][9]
4. **Run repeated samples.** Repeat each important prompt enough times to estimate brand inclusion, citation and position variance separately.[9]
5. **Request raw evidence.** Obtain the original answer, timestamp, prompt, citations and rendered capture where available.[13]
6. **Audit detection manually.** Label true mentions, missed aliases, false matches, parent/sub-brand cases and citation errors.[2]
7. **Recalculate the score.** Verify the numerator, denominator, competitive set, weighting, reporting window and missing-response policy.[10]
8. **Test geography separately.** Compare markets without assuming that a country label controls network location, language, account context and search personalisation.[2]
9. **Check cadence and change logs.** Ask what happens when the model, prompt library, collection method or metric definition changes.[4]
10. **Set acceptance criteria first.** Decide what reproducibility, raw-answer access and reconciliation evidence are required before viewing the results.

Do not set universal pass thresholds without understanding the metric. A 5-point change in a noisy recommendation-position metric may be less meaningful than a smaller but persistent change in brand inclusion. The correct benchmark is the variance baseline for the buyer’s own prompt set and decision.

## Which AI visibility data should a buyer trust?

Trust data that is **specific, inspectable and fit for purpose**. A vendor should explain the measured surface, collection method, prompt source, location and account controls, cadence, sample size, detection logic and score calculation.[2]

The strongest use case for most platforms is directional benchmarking: identifying recurring topics, competitors, citations and changes over time. Ahrefs explicitly cautions that modeled exposure is not actual audience reach, while the IAB says directional measurement is not sufficient by itself for budget allocation or executive strategy decisions.[1]

Decision-grade use requires more:

- Raw answer access.
- Repeat measurements.
- Stable prompt definitions.
- Documented model and platform changes.
- Human-audited detection.
- Clear denominators and weighting.
- Separate reporting for mentions, citations, position and commercial outcomes.

The right question is not “Which vendor has the most accurate score?” It is “Can this vendor demonstrate that its score represents the exact AI surface, population and decision being measured?”

## FAQ
### Are AI visibility tool scores accurate?
They can be accurate descriptions of the responses a platform collected, but they are not automatically representative of every user’s AI experience.[2][3] Accuracy depends on collection surface, prompt sample, context controls, detection quality and aggregation rules.

### Is browser collection more reliable than API collection?
Neither method is universally more reliable. Browser collection may better represent a public consumer interface, while APIs provide explicit configuration; each measures a different surface unless the vendor demonstrates equivalence.[24][11]

### Why do two AI visibility tools report different results?
Tools may use different prompts, engines, model versions, locations, collection methods, cadences, denominators and brand-detection rules.[2] Even identical prompts can return different brands or citations across repeated runs.[3][15]

### How many times should a prompt be run to measure AI visibility?
There is no universal number that applies to every metric. A buyer should repeat important prompts enough to estimate the variance of brand inclusion, citation and position separately, then report confidence or spread rather than relying on one answer.[9]

### Does ChatGPT API data represent what users see in ChatGPT?
Not necessarily. ChatGPT’s consumer search experience can use automatic search, memory and approximate location, while the API uses separately configured search tools and settings.[18][2] A vendor should state which surface it measures.

### How can a buyer test an AI visibility platform’s reliability?
Ask for the exact collection method, model and account context; provide a fixed prompt panel; repeat prompts; inspect raw answers; manually audit mentions and citations; and recalculate the score from its numerator and denominator.[24] Test results should be judged against pre-agreed criteria and a measured variance baseline.

## Sources
[1] Ahrefs Brand Radar Methodology: How we collect and model AI visibility data — https://ahrefs.com/blog/brand-radar-methodology
[2] Bridging Queries and Brands: Entity Linking for Large-Scale Product Search — https://arxiv.org/html/2502.01555v3
[3] The AI visibility data reset: Introducing a new quality standard — https://enterprise.semrush.com/resources/article/ai-visibility-data
[4] My website — https://docs.peec.ai/my-website
[5] Visibility — https://docs.peec.ai/metrics/brand-metrics/visibility
[6] Glossary - Profound — https://help.tryprofound.com/articles/9363559589-profound-glossary
[7] Where does the data in Semrush’s AI Visibility Toolkit come from? — https://www.semrush.com/kb/1607-semrush-ai-visibility-data
[8] AI Visibility Metrics | Help Center - Ahrefs — https://help.ahrefs.com/en/articles/15501968-ai-visibility-metrics
[9] NEW Research: AIs are highly inconsistent when recommending brands or products; marketers should take care when tracking AI visibility - SparkToro — https://sparktoro.com/blog/new-research-ais-are-highly-inconsistent-when-recommending-brands-or-products-marketers-should-take-care-when-tracking-ai-visibility
[10] Web search — https://developers.openai.com/api/docs/guides/tools-web-search?api-mode=responses
[11] Welcome to Peec AI — https://docs.peec.ai/intro-to-peec-ai
[12] Personalization & Google Search results - Google Search Help — https://support.google.com/websearch/answer/12410098?hl=en
[13] Understand how AI is talking about your brand | Profound — https://www.tryprofound.com/features/answer-engine-insights
[14] Profile settings | Perplexity Help Center — https://www.perplexity.ai/help-center/en/articles/10352993-profile-settings
[15] Non-Determinism of “Deterministic” LLM System Settings in Hosted Environments — https://aclanthology.org/2025.eval4nlp-1.12.pdf
[16] Scrunch | FAQs - What methods does Scrunch use to collect data from AI platforms? — https://scrunch.com/faqs/what-methods-does-scrunch-use-to-collect-data-from-ai-platforms
[17] Introduction — https://docs.finseo.ai/
[18] Searching the web with ChatGPT | OpenAI Help Center — https://help.openai.com/en/articles/9237897-chatgpt-search
[19] AI Features and Your Website | Google Search Central  |  Documentation  |  Google for Developers — https://developers.google.com/search/docs/appearance/ai-overviews
[20] Grounding with Google Search  |  Gemini API  |  Google AI for Developers — https://ai.google.dev/gemini-api/docs/google-search
[21] Get personalization in Gemini Apps - Android - Gemini Apps Help — https://support.google.com/gemini/answer/15637730?hl=en&co=GENIE.Platform%3DAndroid
[22] Memory - Perplexity Help Center — https://www.perplexity.ai/help-center/en/articles/10968016-memory
[23] Chat Latest — https://developers.openai.com/api/docs/models/chat-latest
[24] How repeatable are AI recommendations? 4,500 answers from three model APIs — https://askedthrice.com/research/how-repeatable-are-ai-recommendations
[25] Find information in faster & easier ways with AI Overviews in Google Search - Computer - Google Search Help — https://support.google.com/websearch/answer/14901683?co=GENIE.Platform%3DDesktop&hl=en&ref_topic=16011146
[26] Executive Summary — https://www.iab.com/wp-content/uploads/2026/07/IAB_Measuring_Visibility_in_the_AI_Era_August_2026-1.pdf
[27] Data FAQs | Scrunch Help Center — https://helpcenter.scrunchai.com/en/articles/13563748-data-faqs
[28] Setting up your prompts — https://docs.peec.ai/setting-up-your-prompts
[29] How often does OtterlyAI check AI search engines? — https://help.otterly.ai/monitoring-interval
[30] Introducing the Profound Index | Profound — https://www.tryprofound.com/blog/introducing-profound-index
