On this page
Here’s an illustrative example of a conversation we have often: a customer runs their brand through our free Visibility Check and gets 14%. A colleague ran the same brand through a different vendor’s dashboard last month and got 38%. Nobody changed anything about the business in between. The instinct is to assume one tool is broken, or that someone is fudging the number to sell a subscription. Usually neither is true. The two tools asked different questions, checked different engines, counted a “mention” differently, and rolled the result up with different math. Four separate decisions, each reasonable on its own, that can each move the final number by double digits.
This is worth explaining properly, because it comes up constantly once a team is tracking AI visibility from more than one source, and the confusion itself has a cost: people stop trusting the number, which is the opposite of what a scoreboard is for.
”Wrong” is the wrong word
A visibility score is not a single measurement of a single fact, the way a thermometer reads a temperature. It’s the output of a pipeline: a list of prompts, a set of engines those prompts get run on, a rule for deciding whether an answer counts as mentioning you, and a formula for turning hundreds of yes/no results into one percentage. Change any one link in that chain and the number moves, even though nothing about how AI engines actually talk about the brand has changed.
That means two vendors can both be measuring honestly and still land on numbers that do not match, because they are not measuring the same thing. Comparing the headline percentages is a bit like comparing two restaurants’ “rating out of 5” when one averages Google reviews and the other averages Yelp reviews. Both numbers are real. Neither is wrong. They are just not the same rating.
The prompt set is the biggest lever
Your score reflects the specific questions a tool decided to ask. If a vendor’s default prompt set leans toward broad category questions (“best CRM for small teams”) and another leans toward branded and comparison questions (“Salesforce vs HubSpot”), the same company will show up far more often in one set than the other, for reasons that have nothing to do with actual visibility. A brand that dominates comparison questions but is invisible in broad category discovery will look strong on one tool and weak on the other, and both readings are accurate for the prompts each tool chose to ask. Our guide on choosing prompts worth tracking goes into how to build a prompt set that reflects how your actual buyers ask, rather than one that happens to flatter or deflate a score.
Engine coverage changes the denominator
A tool that checks three engines and a tool that checks seven are not producing comparable percentages even off an identical prompt set, because “visible” now means something different in each case. A brand that leads on ChatGPT and Perplexity but has never appeared on Google AI Mode will score high on a three-engine tool that happens to skip AI Mode and lower on a seven-engine tool that includes it. Neither number is inflated or deflated. They are counting a different set of surfaces. Which AI engines to track covers which surfaces actually carry buying-intent traffic right now, which matters more than chasing the vendor with the longest logo list.
A single run is not a measurement
Even holding the prompt set and engine list constant, AI answers are not stable from one run to the next, and this is the part that surprises people most. SparkToro ran 2,961 prompts through ChatGPT, Claude and Google’s AI Overviews and AI Mode using 600 volunteers, and found less than a 1 in 100 chance that the same prompt returns the same list of brands twice. Getting the exact same order twice was closer to 1 in 1,000. The one genuinely reassuring finding in that research is that frequency held up better than order: across nearly a thousand differently-phrased prompts about the same buying intent, the top brands still appeared in 55 to 77% of responses. That’s the case for tracking presence across many runs rather than reading any single answer as the truth, and it’s also why a tool that samples once and reports a precise-looking score is reporting noise dressed up as precision.
The denominator problem
Here’s a worked example that shows how much a single design decision can move a score with nothing else changing. Say a tool tracks 100 answers, 50 of which mention your brand, and only 60 of which mention any tracked brand at all (the other 40 answers didn’t name a single company in the category). Divide mentions by all answers checked and you get 50%. Divide mentions by only the answers that named a brand and you get 83.3%. A September 2026 audit of nine AI visibility platforms found this exact ambiguity inside three of the nine vendors’ own published metric definitions, including one platform whose help documentation used two different denominators for what it called the same metric. Nothing about the brand’s actual answers changed between 50% and 83.3%. Only the arithmetic underneath the label did.
Weighting can swing a score 30 points on its own
Weighting compounds the problem. A September 2026 preprint by researcher Olivier Martinez worked through a case with three prompt categories at fixed, unchanging performance: 51.5% on search-log style queries, 17.4% on retail questions and 94.6% on explanatory prompts. Depending purely on how much weight a tool assigns to each category, the same three underlying rates produce an aggregate score anywhere from 39.7% to 70.5%, a swing of nearly 31 percentage points with not one answer changed. A tool that weights explanatory prompts heavily will report a much rosier number than one that weights retail questions heavily, off the identical set of AI answers.
What to compare instead of the headline number
None of this means scores are useless, it means the single top-line percentage is the least informative part of what a tool gives you. Before comparing your number against a competitor’s dashboard, or last quarter’s number from a different vendor, ask for the parts underneath it:
- The actual prompt list, not just a count of how many prompts it runs
- Which engines are included and which are silently excluded
- How many times each prompt is run before it reports a result
- The exact rule for what counts as a mention, and the denominator it’s divided by
Our own share of voice methodology is built to be checked this way: a fixed prompt set, run across all seven engines we track, with presence logged per answer rather than inferred from a single pass. We’ve written before about why most AI visibility tools stop at a score rather than showing the work behind it, and the same instinct that makes a tool skip diagnosis is what makes it publish a headline number with no methodology page behind it.
Do this instead of arguing over the number
Stop trying to reconcile two vendors’ percentages against each other. It’s the wrong comparison. Pick one tool, keep its prompt set and engine list fixed, and track the trend inside that one system over time, the same way you would not switch analytics platforms every month and expect last year’s traffic numbers to still line up. If you’re evaluating tools, run our free Visibility Check and ask the other vendor for their raw prompt list and mention-matching rule side by side with ours, then decide which methodology you actually trust rather than which score you prefer. And once you have a number you trust, our guide to tracking competitors in AI answers is the more useful next step than re-running the same brand through a fifth dashboard.