What do AI answers and shopper questions measure?
What AI assistants tell shoppers about your store's facts, and whether they name your store when a shopper asks for what you sell.
On its AI answers page, which you open from the Proof page, Emtor asks AI assistants what they tell shoppers, and keeps every answer. The assistants it asks are ChatGPT, Perplexity and Claude. It asks through their developer APIs with web search on, as a shopper in the country from your store facts. Claude is told that country only if it's the US: Anthropic's search refuses countries it doesn't take, so elsewhere Claude searches without one.
Two kinds of question
- About your facts. These name your store and ask about the 2 facts you confirm for it: your return window and your free-shipping amount. Each answer is judged against what you confirmed: it states your fact, states something else, says it doesn't know, doesn't mention it, or is unclear, which never counts as right. They measure whether assistants are right about you.
- What shoppers ask. These never name your store or its domain. They're built from your catalog: your commonest kinds of product, a price most of them fit under, and where you ship. You get 2 or 3 of them. Each answer is read for your store's name or domain, your place in any list it gives, who else it names and what it cites. They measure whether assistants find you.
For a store that sells snowboards in the US, they look like this:
What are the best snowboards under $800?
Where can I buy snowboards online in the US?
Fixed rules read every answer first, and they're tested against sets of phrasings, including ones that must not match. An answer they can't settle, because it gives several figures, a range, a condition or a hedge, is judged by Claude through Anthropic's API, without web search, and the verdict counts only if the answer's own words bear it out. The page shows those words under each verdict.
How often
You start each kind with its own button. You can ask again 24 hours after the last check of the same kind. There's no automatic schedule yet: the one question Emtor asks by itself is the one after a fix, below. Each question goes to each assistant 3 times, because one answer proves little. Each answer is paid for by Emtor, so each store and the whole app have a budget over the last 30 days, which description drafts share. When it's used up, the check doesn't run and the page says so; near the limit, a check can end with fewer answers. Until the check is switched on for your app, the page says so: The AI answers check isn't switched on for this app yet.
When an assistant gets it wrong
Under each question, an assistant that gives another figure than yours, or none, shows the words it said and the page that answer cites, if it cites one. Below them is the fix Emtor has for that fact: its sentence for your refund or shipping policy, added in one click and undone in one; your own edit of the policy, where your store's answer to agents already gives more than one figure; and Correct the fact, for when the assistant is right and the fact you confirmed is out of date. Emtor never changes a confirmed fact itself. The Shipping and returns block and the agent data embed put the same facts on your product pages.
About 24 hours after Emtor adds its sentence, it asks the assistants that question again by itself, once: assistants answer from their own search indexes, which take a while to read a change. It does this only when the check before the fix found an assistant that didn't give your fact, and only while the sentence is still in your policy as Emtor added it. It counts toward the AI budget like any check, and it doesn't hold your own next check back. If the budget can't hold it, it doesn't run, and the page says so. Then the page shows each assistant's right answers before and after the fix. With 3 answers a side, that's a sign, not proof.
What the counts and the interval mean
Results are counts, not percentages: how many of the answers that came back stated your fact or, for shopper questions, named your store. A call that brings no answer isn't counted. With so few answers, chance plays a big part. So every count is worked out with a 95% interval, the range the assistant's true rate plausibly lies in, and the shopper questions show it next to each count. This table gives it for every count:
| Count | Interval |
|---|---|
| 0 of 3 | 0% to 56% |
| 1 of 3 | 6% to 79% |
| 2 of 3 | 21% to 94% |
| 3 of 3 | 44% to 100% |
So 3 of 3 is a good sign, not a guarantee: the true rate could still be under half. A check is an alarm for big changes, not proof of small ones, and the page shows each check on its own, so its range stays this wide.
What the answers are, and aren't
- Close to what shoppers see, not the same. A shopper in an assistant's own app may be signed in, have memory on, or be somewhere else, and can get a different answer.
- Gemini isn't asked. Its API terms don't allow storing and analysing its answers grounded in search, and keeping the answers is the point.
- What leaves your store. The fact questions carry your store's name and domain. Where they go
How each answer is read, and what we don't claim, is on How we measure.