Most Shopify merchants never touch their robots.txt file and are none the worse for it. Shopify's default configuration is sensible, blocks the pages that should be blocked (admin, cart, checkout internals), and leaves your indexable content open to search engines.

The problem arises in two scenarios. First, when a merchant or developer edits robots.txt without understanding what they are changing and accidentally blocks product pages, collection pages, or the entire store. This is not a theoretical risk: Google Search Console commonly surfaces stores where robots.txt has been edited to disallow crawling of product pages, often by a well-intentioned developer who was trying to block duplicate content and went too far.

Second, when a merchant needs to add permissions for AI crawlers, ensure specific paths are allowed, or troubleshoot an indexation problem that robots.txt may be contributing to.

What Does robots.txt Actually Do?

The robots.txt file sits at the root of your domain (yourstore.com/robots.txt) and instructs web crawlers which parts of your site they are allowed to access. It is a text file that follows a simple syntax:

User-agent: Googlebot
Disallow: /admin/
Allow: /products/

User-agent specifies which crawler the rules apply to. Disallow tells the crawler not to access the specified path. Allow explicitly permits access (usually used to override a broader disallow).

Critically: robots.txt controls crawling, not indexing. A page blocked by robots.txt cannot be crawled by that bot, but it might still appear in search results if other sites link to it. Google can index a URL it has never crawled if it has found enough signals about it elsewhere. This is why noindex tags (in the page's HTML) are the correct tool for preventing indexation, while robots.txt is the correct tool for managing crawl access and budget.

For most Shopify merchants, the practical effect is the same: blocking a page in robots.txt prevents Google from reading its content, which prevents it from ranking. But the mechanism is different, and the distinction matters when troubleshooting.

What Does Shopify's Default robots.txt Look Like?

Shopify's default robots.txt blocks crawlers from accessing internal admin and transactional paths that should never appear in search results:

User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /orders
Disallow: /checkouts/
Disallow: /checkout
Disallow: /account
Disallow: /collections/*sort_by*
Disallow: /*/collections/*sort_by*
Disallow: /collections/*+*
Disallow: /collections/*%2B*
Disallow: /collections/*%20*
Disallow: /*/collections/*+*
Disallow: /*/collections/*%2B*
Disallow: /*/collections/*%20*
Disallow: /blogs/*+*
Disallow: /blogs/*%2B*
Disallow: /blogs/*%20*
...
Sitemap: https://yourstore.com/sitemap.xml

This is correct behaviour. Admin and checkout pages have no place in search results. The sort and filter URL patterns are blocked to prevent duplicate content from faceted navigation. The sitemap is referenced, which helps search engines find your content.

What you do not need to do: add disallow rules for product pages, collection pages, or blog posts. These are not in Shopify's default robots.txt, and they should not be.

How to Edit robots.txt in Shopify 2.0

Shopify added the ability to edit robots.txt directly in Online Store 2.0 (available on all current plans). Unlike earlier versions of Shopify where the file was entirely managed by the platform and inaccessible, you can now customise it through your theme.

How to access it:

Online Store > Themes > Actions > Edit Code
In the file list, find templates/robots.txt.liquid (or create it if it does not exist)
Edit the file using Liquid syntax

Shopify's robots.txt.liquid template uses a Liquid object model that loops through the default rules and allows you to add custom rules. The key principle: start with the default rules and add your custom additions, rather than replacing the defaults entirely.

A minimal custom addition to allow a specific path that is being inadvertently blocked:

{% for group in robots.default_groups %}
  {{ group }}
{% endfor %}

User-agent: *
Allow: /collections/sale/

Do not delete the {% for group in robots.default_groups %} block. This renders all of Shopify's default rules. Removing it strips out all the default disallow rules and can cause unexpected indexation of admin pages.

What You Should Never Block

These paths must remain accessible to search engine crawlers:

/products/: Your product pages are the core of your store's organic search value. Blocking these is catastrophic for SEO. Yet this mistake happens: it is common to find stores where a previous developer added Disallow: /products/ while trying to block a specific product, not realising the path rule applies to everything under that prefix.

/collections/: Collection pages are often the highest-traffic SEO pages on a Shopify store. Blocking them removes a major source of category-level organic traffic.

/pages/: Your About page, Contact page, and other static pages provide trust signals and sometimes rank for brand-related queries. These should remain crawlable.

/blogs/: If you have a blog driving organic traffic, these must be crawlable. Blocking them removes all blog-driven SEO value.

Your sitemap URL: The sitemap reference at the bottom of robots.txt should always be present and point to the correct URL. Removing it does not break anything, but it reduces the efficiency of Google's discovery process.

What Is Safe to Disallow

Paths that are safe to add disallow rules for, if not already covered by Shopify's defaults:

/account (already in defaults): customer account pages have no search value
/cart (already in defaults): cart pages should not be indexed
/checkout (already in defaults): checkout pages must never be indexed
Internal search result pages: /search?q= URLs are typically thin pages. Disallowing them is safe and prevents Google from crawling potentially hundreds of dynamically-generated search pages.
Tag-filtered collection URLs not covered by defaults: check your default configuration and add rules for any tag or filter URL patterns generating duplicate content

Allowing AI Crawlers

This is a growing priority for Shopify merchants who want their content cited in AI search engines (ChatGPT, Perplexity, Claude, Grok).

AI crawler bot names to verify are not blocked:

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Shopify's default robots.txt does not block these crawlers, so if you have not edited your robots.txt, they can already access your store. However, if you have added broad disallow rules or have a customised robots.txt from a previous developer, check explicitly that these user-agents are not being blocked.

Google-Extended specifically controls whether your content is used in Google's AI training data and AI Overviews. Disallowing Google-Extended removes your store from Google's AI products. For most merchants, allowing it is the right choice.

If you want to verify which crawlers can access your site, the robots.txt file is public: visit yourstore.com/robots.txt in a browser to see the live file.

Testing with Google's Robots.txt Tester

Google Search Console includes a robots.txt tester at search.google.com/search-console/robots-testing-tool. It shows you the current live content of your robots.txt file and lets you test specific URLs against it.

How to use it:

Enter a URL path you want to test (for example, /products/blue-jacket)
Select the user-agent (Googlebot, or a specific bot)
Click Test: the tool shows whether the URL is allowed or blocked by the current robots.txt rules

This is the most reliable way to confirm whether a specific page is accessible before and after making changes.

After any edit to your robots.txt, test at least three to five representative URLs: a product page, a collection page, your homepage, and any path you have added a custom rule for.

For checking AI crawler access specifically, you can test with custom user-agent names in the tester, though support for non-Google bots varies. The most reliable test for non-Google crawlers is to check the live robots.txt file manually and confirm the relevant user-agents are not disallowed.

Troubleshooting Robots.txt and Indexation Problems

If you suspect robots.txt is causing indexation issues:

Step 1: Open yourstore.com/robots.txt directly and read through the disallow rules. Look for any rules that match your product or collection URL patterns.

Step 2: In Google Search Console, use URL Inspection on a specific product page. If robots.txt is blocking it, the inspection will show "URL is blocked by robots.txt".

Step 3: In Search Console's Pages report, filter for "Blocked by robots.txt" in the Not Indexed reasons. If this shows a large number of pages, you have a misconfiguration.

Step 4: If you find a blocking rule that should not be there, edit robots.txt.liquid in your theme to remove or adjust it. Do not add an Allow rule as a workaround for a problematic Disallow rule: fix the disallow rule itself.

Key Actions to Take Now

Visit yourstore.com/robots.txt in a browser and read through it. Verify there are no disallow rules for /products/, /collections/, or /blogs/.
In Google Search Console, check the Pages report for "Blocked by robots.txt". If any product or collection pages appear here, investigate and fix the blocking rule immediately.
Confirm AI crawlers are not blocked. Check for user-agents GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and Google-Extended. If not mentioned, they are allowed by default.
If you need to edit robots.txt, access it via Online Store > Themes > Edit Code > templates/robots.txt.liquid. Do not delete the default rules block.
After any edit, use Google's robots.txt tester in Search Console to verify affected URLs are allowed.
Test a representative set of page types (product, collection, blog, homepage) against Googlebot in the tester after any change.

Frequently Asked Questions

Can a robots.txt mistake cause my store to disappear from Google? Yes. A Disallow: / rule blocks all crawlers from your entire site. This is an extreme but real mistake. More commonly, Disallow: /products/ or Disallow: /collections/ blocks the most valuable pages. If your store has recently dropped sharply in organic traffic, check robots.txt and Search Console's Pages report immediately.

Should I add a robots.txt rule to block duplicate product variant URLs? No. Duplicate content from product variant URLs (for example, ?variant=12345) is better handled with canonical tags, not robots.txt. Shopify adds canonical tags to variant URLs automatically, pointing them to the main product URL. Adding robots.txt disallow rules for variant URLs can block legitimate crawling in some edge cases.

How do I know if an AI crawler is already accessing my site? Your server logs will show crawl activity by user-agent. If you do not have access to raw server logs (most Shopify merchants do not), you can infer from Cloudflare Analytics or a WAF if you use one. Practically speaking, if your robots.txt does not disallow the known AI crawler user-agents, they can access your site.

Does robots.txt affect how quickly new products get indexed? Not directly. Robots.txt controls whether crawlers can access pages, not how quickly they crawl them. New product indexation speed is affected by crawl budget (how often Google revisits your site), internal linking to the new product, and whether you have submitted an updated sitemap. To speed up indexation of a specific new product, use the URL Inspection tool in Search Console to request indexing directly.

Shopify Robots.txt: What It Does, How to Edit It, and What Not to Block