In short
- 20 online shops in Switzerland (12), Germany (5) and Austria (3).
- An openly identified agent read each shop's public pages the way many AI assistants do: plain HTML, no JavaScript, respecting robots.txt.
- It looked for answers to the same 10 questions shoppers ask before and after buying.
- Failed Intent Rate (FIR) = questions the site could not answer ÷ questions asked. Lower is better.
- Every unanswered question was checked by hand before it counted.
The 10 questions
Costs, delivery time, returns and payment methods are among the most common reasons shoppers abandon a purchase. Invoice payment is a specific expectation in the DACH market. The agent searched with German and English keywords.
| # | Question | ID |
|---|---|---|
| 1 | How many days do I have to return an item? | returns-window |
| 2 | Are returns free, or do I have to pay for return shipping? | returns-cost |
| 3 | How much does shipping cost, and from what order value is it free? | shipping-cost |
| 4 | How long does standard delivery take? | delivery-time |
| 5 | Which countries do you deliver to? | countries |
| 6 | Which payment methods do you accept? | payment-methods |
| 7 | Can I pay by invoice (Kauf auf Rechnung)? | invoice |
| 8 | How can I reach customer service, and when are they available? | contact |
| 9 | How do I track my order? | track-order |
| 10 | How long is the warranty, and how do I make a warranty claim? | warranty |
How the agent read each site
- User agent:
fir-audit/0.1 (+https://github.com/lurus92/fir-audit). It never pretended to be a browser. The code repository in the user agent opens to the public on 20 November 2026; until then, this page is the reference. Future runs identify asfir-audit/0.1 (+https://lrusso.it/fir/method/). - robots.txt respected, including Crawl-delay. At most one request per second, one site at a time.
- Same-site pages only, up to 40 pages per site. Links matching the questions' keywords (returns, shipping, payment, contact…) were fetched first.
- Static HTML only: no JavaScript was run, no forms submitted, no login, no chat widgets.
- Content crawl: 22 October 2026. Access re-checked on 21, 22 and 24 October 2026.
Access: can an agent get in at all?
Before scoring content, each shop was placed in one of these groups:
- Scored: at least 5 help or policy pages with readable text in the HTML.
- Blocked: the server refused the agent (HTTP 401, 403 or 429) in the majority of at least 4 checks over 3 days.
- Inconsistent: refused in some checks and let in during others. Reported as such, not scored and not counted as blocked.
- Help not readable without JavaScript: the site responded, but fewer than 5 help or policy pages had their text in the HTML.
- Unreachable: network or server errors. This says nothing about a shop's policy.
For every shop the study also records what robots.txt says to 14 known AI crawlers and assistants (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Amazonbot, CCBot, Bytespider). That makes it possible to compare what a site says it allows with what its server actually does.
How a question counts as answered
- For each question, the 5 most relevant pages the agent read are selected (language versions of the same page count once).
- A language model (gpt-4o-mini, temperature 0) reads only those excerpts, with no outside knowledge of the company, and decides whether a shopper could get a specific, usable answer.
- An answer only passes if the model quotes the page word for word and the quote is found on the page. "Contact us for details" is not an answer.
- Every question marked unanswered is then checked by hand. Only failures confirmed by a person count.
Validation
- 60 of the 100 scored question-and-shop results were reviewed by hand, including every failure.
- Of 54 passes checked by hand, 2 were wrong (3.7%). The remaining passes rely on the model with a verified quote, so FIR may be slightly understated, never overstated.
- A second model used as a "verifier" was tested and dropped: it disagreed with human review too often.
Limits
- One IP address, one user agent, four days of checks. A firewall may treat other agents differently.
- Static HTML only. Many AI assistants read pages this way, but some run JavaScript. Sites whose help centre needs JavaScript are reported, not scored.
- 40 pages per site is a sample. An answer that exists deeper in a site can be missed. That is what the correction step is for.
- Two-part questions (warranty length and how to claim) make "answered" harder to judge. They will be split in the next version.
- With 10 scored shops, a difference of one or two questions is not a meaningful ranking.
Corrections and right of reply
Before anything is public, every shop receives its own result by email. If a result is wrong, for example a page the agent missed, or a change made after 22 October 2026, reply by 13 November 2026 and the shop is re-tested and corrected before publication. Shops can also send a short statement to be included in the study.
If your team needs the details of a specific check (timestamps, pages read, the excerpt and quote behind each verdict), write to [email protected].
Timeline
- 21–24 October 2026: access checks and content crawl.
- 13 November 2026: deadline for corrections and statements.
- 20 November 2026: the test tool is released as open source.
- 21 November 2026: results presented at MeasureCamp Zurich and published.
Who runs this
Luigi Russo, an analytics lead based in Zurich (previously TikTok, Vodafone and Sunrise), who coined the Failed Intent Rate metric. Disclosure: he also builds AskVio, an assistant for websites. The study is independent of AskVio, and a shop's result does not depend on using it.