PDP-A03AI search & agents · agentMedium

llms.txt present at root; product URLs not bot-blocked

llms.txt present at root; product URLs not blocked by robots.txt for AI crawlers.

Check my store for this rule →Cite this

Free · no card · runs all rules at once · judged on 40 stores · updated 23 Sep 2026

3%of the audited stores we could judge fail this rule

30 pass9 partial1 failPublished audits · September 2026

Stores judged

of 40 published audits

Score weight

7.7pts

of the AI Shoppers score

Audience

reported separately

What earns each verdict

Same outcomes on every UXFix rule · how verdicts work

✓ Pass30

/llms.txt exists (any content) AND robots.txt does not block AI crawlers from product URLs.

~ Partial9

robots.txt clear but no llms.txt (llms.txt is optional but signals intent).

✕ Fail1

AI crawlers explicitly disallowed from product paths.

not assessable is recorded when the rule applied but the store could not be judged on it (bot protection, a region lock, a step we could not reach), and not applicable when the store has nothing for the rule to judge. Those stores are left out of every number on this page, never guessed. Why we do that

Why it matters

The site does not publish an llms.txt at its root, or its robots.txt actively blocks AI crawlers like GPTBot, ClaudeBot, or Google-Extended from your product pages.

llms.txt is the emerging standard for telling AI agents what your site offers, similar to sitemap.xml for search engines. Blocking AI crawlers means you opt out of appearing in ChatGPT and Google AI answers. That is a growing traffic source.

That sentence is the one a failing store sees in its own UXFix report.How rules are weighted and scored →

40 audited stores, one dot each

1 fail

PassPartialFail

Every published UXFix audit this rule could be judged on, as of 23 Sep 2026. A count of real stores, not a simulation of yours.

How we test it

Scripted walk, fresh signed-out browser · the full walk

1Fresh browserSigned out, cookies cleared, desktop and mobile viewports, the store's live theme.
2Open a productA real, in-stock product page, found the way a shopper finds one.
3Read it like an agentThe markup and structured data a shopping agent reads, not the layout.
4Record the verdictPass, partial or fail, saved with the screenshot it was judged on.

What the agent looks for

llms.txt present at root; product URLs not blocked by robots.txt for AI crawlers.

Confidence

The rulebook rates this check’s confidence High. When the agent is not sure, the rule is recorded as not assessable rather than guessed.

Not judged

0 of 40 published audits: the rule did not apply or could not be judged. Dropped, never guessed.

The numbers behind the rule

Published UXFix audits · more statistics

Published audits failing this rule

1Davidsbridaldavidsbridal.com58

No pictures here: this rule is judged inside the page code, which no shopper ever sees, so there is nothing to photograph.

30 stores that do it right →All published audits

Where this rule sits among 12 agent rules

Most-failed rank

#77

Score weight rank

Fail rate

Score weight: a fail costs 7.7 pts of of the AI Shoppers score; a partial costs half that. Most-failed rank is among the 79 rules with enough published audits to report a rate.

The fix

The same steps a failing store sees in its report

A pass means: /llms.txt exists (any content) AND robots.txt does not block AI crawlers from product URLs.

Severity

Medium

Score gain

+7.7pts

Page

AI search & agents

Score gain is roughly what a fail-to-pass change adds; rules that could not be judged are left out, which can move it a little. Re-run a free audit after the change ships to confirm the verdict moved.

Checklist for the pass

0 of 3

Tick each step as you ship it. The audit judges the live store, so re-run it once all of them are done.