Why AI Shopping Agents Can't Find Your Product Pages

If ChatGPT, Gemini or Copilot can't recommend your product page, the most common reason is that they never crawled it in the first place. AI shopping agents rely on the same fetch-and-render pipeline as search engines: a robots.txt that allows their user-agent, a sitemap that lists the URL, and HTML that doesn't require JavaScript to show price and availability. Miss any one of those and the agent skips the page entirely.

The crawlability gap nobody checks

Most store owners we talk to have already thought about schema markup and checkout friction. Almost none have asked whether an AI agent can actually fetch and render their product page in the first place. That's a step earlier than schema, and it's the one that determines whether schema even gets read.

This matters because AI shopping agents don't browse the way a person does. They send a request, get back raw HTML, and decide from that response whether the page is worth citing. If the response is blocked, empty, or points somewhere else via a bad canonical tag, the agent moves on.

Here's a worked example we see often. A mid-sized apparel store has clean Product schema, correct pricing, and a fast page. But their robots.txt was written three years ago for a security audit and disallows any user-agent containing the word "bot" that isn't explicitly Googlebot or Bingbot. GPTBot matches that pattern. The schema is perfect and irrelevant, because the page is never fetched.

The same gap shows up further down the funnel. Once an agent (or a person) reaches checkout, a different set of failures takes over, and we've covered those separately in 8 rules for fixing checkout abandonment at the payment step. Crawlability is the filter that decides whether a shopper, human or AI, ever gets that far.

Why can't AI shopping agents find my product pages? Seven technical blockers

In practice, the failures cluster around a small set of causes. This is the crawlability checklist we run against every product page during a UXFix audit, roughly in the order we check them.

• robots.txt disallows the agent. Rules written for Googlebot don't automatically cover GPTBot, ClaudeBot, PerplexityBot or Google-Extended. A wildcard disallow rule, or one written against an outdated list of "known bots", is the single most common cause we log.
• The URL is missing from the sitemap. No entry means no discovery path for a fresh page. This is especially common with seasonal SKUs added mid-catalog after the sitemap generator last ran.
• Canonical tags point elsewhere. A PDP that canonicalizes to a category page, or to a different colour variant, tells crawlers to ignore the exact URL a shopper or agent is looking for.
• Price and stock load via JavaScript only. The first HTML response has no price, so there's nothing to cite. We cover why this also hurts human shoppers in 12 fixes for shoppers who leave your product page without buying.

The next three are less obvious, but just as common in the audits we run.

• Infinite scroll hides the PDP. If a product only appears after a scroll or click event with no static URL behind it, there may be nothing for a crawler to request at all.
• Noindex tags left over from staging. Common after a redesign, and easy to miss because the page still renders correctly for a person clicking through the live site.
• Redirect loops or errors on filtered URLs. Faceted navigation can quietly break the exact variant URL a shopper searched for, sending both agents and search crawlers into a 302 loop or a soft 404.

Any one of these is enough to keep a page out of an AI agent's index. Several of them together explain why a store with solid schema still gets zero AI-driven traffic, and why fixing schema first is usually the wrong order of operations.

An edge case worth knowing: partial blocking

Some of the worst cases we've audited aren't a full block. They're a partial one. A CDN or web application firewall blocks a crawler's IP range during traffic spikes, while robots.txt itself allows the user-agent.

From the store's side, everything looks fine: the robots.txt file reads correctly, the sitemap is current, and a manual browser check shows the page rendering. But the agent's actual request is dropped at the network layer before it reaches the origin server. This is why we recommend testing with a real fetch, not just reading configuration files.

robots.txt rules for AI agents

Robots.txt is still the first gate. If you already allow Googlebot, you need a deliberate decision about whether to allow the AI crawlers too, not a default block left from a security template.

User-agent: Googlebot
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://example.com/sitemap.xml

That's the shape of a robots.txt file that makes a deliberate choice rather than an accidental one. Compare it against your own file line by line. A single leftover Disallow: / block under a catch-all User-agent: * rule will override every specific Allow line beneath it, depending on how the parser resolves conflicts, so specificity matters more than most teams assume.

Google's own crawling and indexing documentation is a useful baseline here, since most AI crawlers follow the same robots.txt conventions Google helped standardize.

Does my product page get indexed by AI search?

Crawling and indexing are not the same thing, and citation is a third step again. A page can be fetched, fail to render correctly, and never make it into an index an agent draws from.

The most common break point we see is client-side rendering. If price, stock status and variant options only appear after JavaScript executes, the raw HTML an agent receives on first fetch can be nearly empty.

Worked example: a home goods retailer moved their product detail template to a client-side framework two years ago for a smoother variant switcher. The visual experience for a person is fine, price updates instantly when you pick a different size. But the initial document a crawler requests contains a loading skeleton and an empty price field, populated later by a fetch call to an internal API. Every AI crawler that doesn't execute JavaScript, and several that do but hit a timeout, see a page with no price.

This is well documented outside of AI search too. Google's guidance on rendering and Baymard's usability research both point to the same underlying problem: content that depends on scripts to appear is content that's easy to miss, whether the visitor is a person or a bot. The same rendering delay that hides a price from a crawler is often the exact delay that makes a mobile shopper bounce before the page finishes loading, a pattern we detail in 9 UX mistakes causing mobile checkout abandonment.

How to test whether an AI agent can see your page

You don't need to wait for an audit to check this yourself. A basic fetch from the command line shows you exactly what a crawler that doesn't execute JavaScript would see.

curl -A "GPTBot" -s https://example.com/products/your-product | grep -i "price"

If that command returns nothing, or returns a loading placeholder instead of an actual number, you've found the same blank response an AI agent gets. Run the same check against robots.txt directly:

curl -s https://example.com/robots.txt | grep -A2 "GPTBot"

An empty result usually means no rule exists for that user-agent, which typically falls back to whatever the wildcard User-agent: * block says. That fallback is where most accidental blocks live.

A few edge cases worth checking alongside this:

• Regional or currency redirects. If visiting the base URL triggers a redirect based on IP geolocation, a crawler's data-centre IP may get routed to the wrong regional site, or into a redirect loop if the target region doesn't match any configured rule.
• Bot-detection challenges. Some fraud-prevention tools serve a CAPTCHA or a JavaScript challenge page to any request without a browser fingerprint. That's correct behaviour for blocking scrapers, but it can catch legitimate AI crawlers in the same net.
• Rate limiting during traffic spikes. A crawler making sequential requests across a large catalog can trip rate limits meant for abusive scraping, especially during a flash sale when your own traffic is already elevated.

Building an ecommerce sitemap for AI shopping assistants

Your sitemap is the discovery mechanism. If a product URL isn't listed, an agent has to stumble onto it through internal links, which is unreliable for a large catalog.

A sitemap built for AI shopping assistants should:

• List every canonical PDP URL, not category or filtered variants.
• Update lastmod accurately when price or stock changes, so agents know to re-fetch.
• Exclude discontinued or out-of-stock-forever products rather than leaving dead URLs in place.
• Stay under the size limits search engines already enforce, since most AI crawlers reuse the same sitemap protocol.

<url>
  <loc>https://example.com/products/your-product</loc>
  <lastmod>2026-01-14</lastmod>
  <changefreq>daily</changefreq>
</url>

None of this requires new infrastructure. It requires treating the sitemap as a live feed rather than a file generated once at launch and forgotten. Stores that regenerate their sitemap only on deploy, rather than on a schedule tied to inventory changes, are the ones most likely to have stale lastmod values that give a crawler no reason to re-fetch a page whose price changed yesterday.

It's worth noting that sitemap and crawlability problems compound with problems further down the funnel. A page an agent can find and cite is only useful if the account creation step or the payment step doesn't then lose the shopper it sent you, which is the subject of why shoppers abandon checkout at account creation.

Questions we get asked