Ctrlk
Back to PageFly.ioVideo Tutorials
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Your robots.txt is blocking AI bots from reading your pages

Your robots.txt is blocking AI bots

Shoppers now ask ChatGPT, Claude, Perplexity and Google for product recommendations. Those assistants can only recommend pages they are allowed to read. If PageFly tells you your robots.txt is blocking AI bots, this guide explains what that means and what to do.

What you are seeing

Run an AEO scan from Your store today on your PageFly Dashboard. Under the score you may see:

Your robots.txt is blocking some AI bots from reading this page.

If you scanned a page that is not yours, the wording is slightly different:

This site's robots.txt is blocking some AI bots from reading this page.

If AI bots are allowed, this line does not appear at all. No message is good news.

What robots.txt is

Every website has a file at /robots.txt. It is a public list of instructions that tells automated visitors which parts of the site they may read. Search engines and AI assistants check it before they crawl anything.

Shopify creates this file for you automatically. It only becomes a problem when a rule inside it tells AI assistants to stay away.

Which AI bots PageFly checks

Name in robots.txt
Who it belongs to

GPTBot

OpenAI. Reads pages for ChatGPT.

OAI-SearchBot

OpenAI. Powers search and browsing inside ChatGPT.

ClaudeBot

Anthropic. Reads pages for Claude.

PerplexityBot

Perplexity.

Google-Extended

Google. Controls whether your content can appear in AI answers such as AI Overviews and Gemini.

*

A wildcard. A rule here applies to every bot, including all of the above.

Why this matters

AEO is about being found and quoted by AI assistants. Everything else you do for AEO, such as adding an FAQ, structured data, clear headings or a spec table, is wasted on a page an assistant is not allowed to open. This is the first door. If it is shut, nothing behind it counts.

Check it yourself

Open this in your browser, with your own domain:

You will see a plain text file. Look for a block that names one of the bots above, followed by a line starting with Disallow:.

A block like this stops ChatGPT from reading your entire store:

When this is normal and you should do nothing

Shopify's default robots.txt blocks a few system URLs from every bot, such as your cart, your checkout and your internal search results. That is deliberate and correct. Those pages are not meant to be read by anyone but a shopper.

So if you scanned a cart, checkout or search URL and saw the message, there is nothing to fix. Scan a product page, a collection page or a landing page instead.

How to fix it

These rules live in a theme file called robots.txt.liquid. PageFly cannot change it for you, which is why this notice links to a guide instead of offering a one-click fix.

Shopify's warning, which we repeat. Editing this file is an unsupported customization. Shopify Support cannot help with edits to it, and incorrect use can result in loss of all traffic. If you are not comfortable editing theme code, ask your developer, or contact PageFly support and we will look at your robots.txt with you.

  1. In your Shopify admin, go to Online Store > Themes.

  2. Find your live theme, open its ... menu and choose Edit code.

  3. In the Templates folder, look for robots.txt.liquid.

If the file is not there, nothing on your store has been customized and your robots.txt is Shopify's default. Shopify's default does not block GPTBot, ClaudeBot or the others. What you saw is almost certainly the normal case described above, so scan a product or landing page instead.

If the file is there, someone has customized it, usually an agency, a developer, or an SEO app. Continue:

  1. Read the file and find the rule that either names one of the AI bots, or uses * and disallows the page you scanned.

  2. Remove that rule, or narrow it so it no longer covers the pages you want AI assistants to read.

  3. Click Save.

If an SEO app added the file, change the setting inside that app instead of in the code. Otherwise the app may write its rule back the next time it runs.

Shopify's own instructions: Edit robots.txt.liquid

After you fix it

  1. Open https://yourstore.com/robots.txt again and confirm the rule is gone.

  2. Go back to your PageFly Dashboard and rescan the page. Scans have a short cooldown, so wait a moment if the button does not respond straight away.

  3. The message disappears once AI bots are allowed.

Your AEO score will not go up, and that is expected. This notice is a warning, not a deduction. What changes is that assistants can now actually reach the page your score describes.

FAQ

Does unblocking them mean AI will start recommending my products?

No. It means they are allowed to look. Whether they quote you depends on what is on the page. Run an AEO scan and work through the findings it gives you.

I want search engines in but AI out. Can I do that?

Yes, that is exactly what the separate bot names are for. Blocking GPTBot does not affect Google's normal search crawling. Google-Extended is the one worth understanding: it controls AI answers only, so blocking it does not remove you from ordinary Google search results.

Does PageFly change my robots.txt?

No. PageFly only reads it while a scan is running.

The message appears for a competitor's URL I scanned.

That is their setting, not yours, and you cannot change it. It is still useful to know: their pages may not be appearing in AI answers either.

I removed the rule but the message is still there.

Check that you edited your live theme and not a duplicate, then reload /robots.txt to confirm the change is public. Rescan after a minute.

Last updated

Was this helpful?