> For the complete documentation index, see [llms.txt](https://help.pagefly.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://help.pagefly.io/page-structure-and-elements/optimization/robots-txt-blocking-ai-bots.md). # Your robots.txt is blocking AI bots from reading your pages ## Your robots.txt is blocking AI bots Shoppers now ask ChatGPT, Claude, Perplexity and Google for product recommendations. Those assistants can only recommend pages they are allowed to read. If PageFly tells you your robots.txt is blocking AI bots, this guide explains what that means and what to do. ### What you are seeing Run an AEO scan from **Your store today** on your PageFly Dashboard. Under the score you may see: > Your robots.txt is blocking some AI bots from reading this page. If you scanned a page that is not yours, the wording is slightly different: > This site's robots.txt is blocking some AI bots from reading this page. If AI bots are allowed, this line does not appear at all. No message is good news. ### What robots.txt is Every website has a file at `/robots.txt`. It is a public list of instructions that tells automated visitors which parts of the site they may read. Search engines and AI assistants check it before they crawl anything. Shopify creates this file for you automatically. It only becomes a problem when a rule inside it tells AI assistants to stay away. ### Which AI bots PageFly checks | Name in robots.txt | Who it belongs to | | ------------------ | ----------------------------------------------------------------------------------------------- | | `GPTBot` | OpenAI. Reads pages for ChatGPT. | | `OAI-SearchBot` | OpenAI. Powers search and browsing inside ChatGPT. | | `ClaudeBot` | Anthropic. Reads pages for Claude. | | `PerplexityBot` | Perplexity. | | `Google-Extended` | Google. Controls whether your content can appear in AI answers such as AI Overviews and Gemini. | | `*` | A wildcard. A rule here applies to every bot, including all of the above. | ### Why this matters AEO is about being found and quoted by AI assistants. Everything else you do for AEO, such as adding an FAQ, structured data, clear headings or a spec table, is wasted on a page an assistant is not allowed to open. This is the first door. If it is shut, nothing behind it counts. ### Check it yourself Open this in your browser, with your own domain: ``` https://yourstore.com/robots.txt ``` You will see a plain text file. Look for a block that names one of the bots above, followed by a line starting with `Disallow:`. A block like this stops ChatGPT from reading your entire store: ``` User-agent: GPTBot Disallow: / ``` ### When this is normal and you should do nothing Shopify's default robots.txt blocks a few system URLs from every bot, such as your cart, your checkout and your internal search results. That is deliberate and correct. Those pages are not meant to be read by anyone but a shopper. So if you scanned a cart, checkout or search URL and saw the message, there is nothing to fix. Scan a product page, a collection page or a landing page instead. ### How to fix it These rules live in a theme file called `robots.txt.liquid`. PageFly cannot change it for you, which is why this notice links to a guide instead of offering a one-click fix. > **Shopify's warning, which we repeat.** Editing this file is an unsupported customization. Shopify Support cannot help with edits to it, and incorrect use can result in loss of all traffic. If you are not comfortable editing theme code, ask your developer, or [contact PageFly support](https://help.pagefly.io/getting-started/quick-start/how-can-i-access-the-pagefly-live-chat.md) and we will look at your robots.txt with you. 1. In your Shopify admin, go to **Online Store** > **Themes**. 2. Find your live theme, open its **...** menu and choose **Edit code**. 3. In the **Templates** folder, look for `robots.txt.liquid`. **If the file is not there**, nothing on your store has been customized and your robots.txt is Shopify's default. Shopify's default does not block GPTBot, ClaudeBot or the others. What you saw is almost certainly the normal case described above, so scan a product or landing page instead. **If the file is there**, someone has customized it, usually an agency, a developer, or an SEO app. Continue: 4. Read the file and find the rule that either names one of the AI bots, or uses `*` and disallows the page you scanned. 5. Remove that rule, or narrow it so it no longer covers the pages you want AI assistants to read. 6. Click **Save**. If an SEO app added the file, change the setting inside that app instead of in the code. Otherwise the app may write its rule back the next time it runs. Shopify's own instructions: [Edit robots.txt.liquid](https://help.shopify.com/en/manual/promoting-marketing/seo/editing-robots-txt) ### After you fix it 1. Open `https://yourstore.com/robots.txt` again and confirm the rule is gone. 2. Go back to your PageFly Dashboard and rescan the page. Scans have a short cooldown, so wait a moment if the button does not respond straight away. 3. The message disappears once AI bots are allowed. Your AEO score will not go up, and that is expected. This notice is a warning, not a deduction. What changes is that assistants can now actually reach the page your score describes. ### FAQ **Does unblocking them mean AI will start recommending my products?** No. It means they are allowed to look. Whether they quote you depends on what is on the page. Run an AEO scan and work through the findings it gives you. **I want search engines in but AI out. Can I do that?** Yes, that is exactly what the separate bot names are for. Blocking `GPTBot` does not affect Google's normal search crawling. `Google-Extended` is the one worth understanding: it controls AI answers only, so blocking it does not remove you from ordinary Google search results. **Does PageFly change my robots.txt?** No. PageFly only reads it while a scan is running. **The message appears for a competitor's URL I scanned.** That is their setting, not yours, and you cannot change it. It is still useful to know: their pages may not be appearing in AI answers either. **I removed the rule but the message is still there.** Check that you edited your live theme and not a duplicate, then reload `/robots.txt` to confirm the change is public. Rescan after a minute. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://help.pagefly.io/page-structure-and-elements/optimization/robots-txt-blocking-ai-bots.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.