Most Shopify stores can leave robots.txt alone. Change it only when you can name the URLs or crawlers that need a different policy—and show that the change leaves important storefront content accessible.
Adding robots.txt.liquid is more than adding one line to a file. It replaces Shopify's generated output, and the Liquid defaults can differ from the managed file. A small intended edit can therefore introduce unrelated changes.
These 11 Shopify robots.txt checks help you decide whether to edit, define the exact effect and prepare a recovery plan. They draw on current Shopify and Google documentation; the examples are illustrative, not results from testing a merchant's store. They cover Shopify's hosted Online Store. For a headless storefront, use the separate Shopify Hydrogen SEO checks and inspect the implementation serving that site.
1. Choose the right fix for the actual problem
Write down the outcome before writing a directive. “Stop crawling these low-value URL variations” is a different task from “remove this page from Google.”
| Your goal | What to investigate |
|---|---|
| Reduce requests to a specific URL pattern | A narrowly scoped crawl rule, after checking the URLs' value and current treatment |
| Keep a public page out of Google results | A supported noindex implementation that Google can fetch |
| Protect customer-only or confidential content | Authentication or another real access restriction |
| Recover an important page reported as blocked | The rule that actually matches it, alongside the current page response |
Google's robots.txt introduction explains why a crawl block is not reliable index removal: a URL can still be discovered through links. Blocking also prevents Google from reading a noindex directive on that page.
For example, adding Disallow: /products/ to hide a handful of discontinued products can prevent fetching of active products too. Decide how those individual products should behave first; do not turn a page-level decision into a store-wide rule.
Keep the change request to one sentence: “Prevent crawler X from fetching URL family Y, while keeping Z accessible.” If you cannot complete that sentence, gather evidence before editing.
2. Capture the live file on the correct domain
Start at https://your-primary-domain.example/robots.txt, not a file copied from another store or an old theme. Record the hostname, time, response headers and complete body.
This read-only command saves a baseline locally. Replace example.com with your own storefront hostname; on Windows, use curl.exe if curl resolves to a different command.
curl --silent --show-error --dump-header robots-before.headers --output robots-before.txt https://example.com/robots.txt
The command deliberately does not follow redirects. Inspect a redirect's destination before fetching it separately. A successful download alone does not establish that the body is valid robots.txt: check the status, content type and text for an error page, challenge or unexpected content.
Google scopes robots.txt to its host, protocol and port. A capture from a myshopify.com address or another market domain is not evidence of the primary custom domain's policy.
Keep this response alongside the existing custom template, if one exists. They answer different questions: the response shows the rules being served, while the template shows how they are produced.
3. Find the published theme and the file's owner
In Shopify admin, open Online Store → Themes and identify the published theme. Its custom template, when present, belongs at templates/robots.txt.liquid. It is not an asset file or a JSON template.
Before changing it, establish who maintains it. Check the template's comments and recent changes, and ask whether an app or agency manages those rules. If a previous setup rewrites the response outside Shopify, coordinate that setup's removal or migration with its owner. Do not layer a second implementation over an unexplained first one.
A missing template does not mean robots.txt is missing: Shopify serves a generated file without one. Editing an unpublished theme also does not prove the public file has changed.
The Shopify template reference documents the correct location and output behavior. Include the file in the next theme-change review, too. Shopify's Help Center notes that uploading a theme through the admin's Themes section does not import robots.txt.liquid; command-line changes preserve it.
Record the theme name and the responsible person or team. After a theme replacement, verify the root response again rather than assuming the custom policy survived.
4. Compare managed defaults with Liquid output
Do not assume that a template looping over robots.default_groups reproduces the managed file byte for byte.
Shopify's current documentation makes two points that need to be read together:
- Use the provided Liquid objects where possible rather than freezing a plain-text copy.
- The rendered template can differ from Shopify's generated output. Shopify specifically notes that template defaults can include
Disallow: /searchandDisallow: /policies/after those rules have disappeared from the managed file.
The current customization guide therefore recommends saving the existing response and comparing it with the rendered template before publication.
Review the entire diff, including crawler groups, existing Allow exceptions, sitemap lines and rules you did not intend to change. “Added one Disallow” is not an adequate description if creating the template also changes unrelated paths.
Retain the Liquid structure when customization is justified, but treat its output as something to verify. Do not automatically remove /search or /policies/ blocks solely because they appear in an older file; inspect your store's page-level policy and requirements.
If the custom template has no continuing purpose, consider returning to Shopify's managed output. Save the current template first, and approve that resulting policy before deleting anything. Check 11 explains the difference between restoring a previous customization and resetting to the platform default.
5. Protect product, collection and rendering URLs
Build a small set of URLs that must remain fetchable. Include examples that your proposed pattern might accidentally catch, not just the homepage.
| URL role | What to preserve |
|---|---|
| Active product | Access to its description, offer and page signals |
| Main collection | Access to the intended category landing page |
| Next collection page | Discovery paths to products beyond the first page |
| Useful market landing page | Access to the correct language or regional content |
| Required image, CSS or JavaScript | Resources needed to render and understand the page |
| App-generated public page | Access where that page has an intended public purpose |
These are acceptance criteria for your own store, not a claim that every URL variant should be indexed.
Avoid blanket blocks on /products/, /collections/, /cdn/ or all query strings. A filter, product variant and pagination parameter can have different jobs even though each produces a URL containing ?.
For filtered collections, choose the intended landing pages before adding a broad rule. Our guide to Shopify product tag mistakes explains why public filters and operational tags need different treatment.
Run this protected set against the proposed full file. A rule that catches the unwanted URL and a valuable collection has failed the check.
6. Test the boundaries of every proposed pattern
Write at least one URL that should match and one similar URL that must not match each new rule.
Google's path matching is case-sensitive. Rules normally match prefixes; * can match a sequence of characters, and $ anchors the end. A broad pattern can therefore reach farther than its author intended.
Suppose a hypothetical store wants to block crawling of one retired campaign collection. This isolated example illustrates the boundary; it is not a recommendation to block that collection on your store:
Disallow: /collections/retired-campaign$
| Illustrative URL path | Matches this single rule? |
|---|---|
/collections/retired-campaign |
Yes |
/collections/retired-campaign?sort_by=title-ascending |
No: the query continues after the anchored path |
/collections/retired-campaign-new |
No |
/en/collections/retired-campaign |
No |
/collections/Retired-Campaign |
No |
Removing $ broadens the match, including the longer campaign handle. If parameterized URLs are also in scope, specify and test them deliberately. Include alternate parameter orders and encoded forms that your actual storefront generates.
Finally, evaluate the complete file. Google uses the most specific matching path and the less restrictive rule when conflicting rules tie. A single-line match table cannot establish the final allow/block decision when other rules or crawler groups apply.
7. Evaluate the crawler group that actually applies
An extra crawler group can change which existing rules apply. For Google's crawlers, a more specific user-agent group takes precedence over User-agent: *; the two are not inherited or combined.
Consider this deliberately simplified robots.txt example:
User-agent: *
Disallow: /collections/retired-campaign
User-agent: Googlebot
Disallow: /pages/internal-promotion
Googlebot uses its named group. The retired-campaign block in the wildcard group does not apply to Googlebot through inheritance. Adding a named group “for one extra rule” has therefore changed more than one permission.
Google also combines rules from multiple equally specific groups for the same agent. Moving a duplicate group lower in the file does not make it override the earlier one.
Check the final response for all occurrences of the agent name, then test your URL set with that agent selected. Consult Google's robots.txt specification for its matching behavior. For another crawler, verify that crawler's own documentation rather than assuming identical treatment.
When editing Shopify's Liquid, identify the intended group explicitly—for example, its group.user_agent.value—and inspect where the resulting line lands. A correct path in the wrong group is still the wrong policy.
8. Check market domains and localized paths separately
A rule checked on one storefront origin has not been checked on every market.
List the domains and subdomains you actually use, then capture each one's root /robots.txt. For a store using language subfolders, add localized examples to the URL set as well. The illustrative rule in check 6 does not match /en/collections/retired-campaign.
Shopify's customization guide supports request.host for host-specific rules when distinct Markets domains or subdomains genuinely need different crawl behavior. That is a precise condition, not a reason to create a different policy for every market.
For example, a team may need one domain-specific exclusion while retaining the other domain's public collection pages. Record the exact host and path combination, then inspect the rendered response on both hosts. A condition copied into the template without checking its output is insufficient.
Do not block useful translated collections simply because they sell the same products. First decide the intended regional page, its canonical and language relationships. If one shared rule should cover several localized paths, document each expected match rather than assuming a root-path rule covers them all.
9. Read page signals and sitemap destinations together
For each affected URL family, inspect a representative page's actual response as well as its robots permission. Record the HTTP status, canonical, meta robots and any X-Robots-Tag header.
This turns a vague exclusion request into a concrete diagnosis:
- Blocked URL carrying noindex: Google cannot fetch the page to read that instruction. Resolve the crawl restriction if relying on noindex for removal.
- Blocked alternate URL carrying a canonical: Google cannot fetch the page to read that canonical. Blocking and canonicalization are separate decisions.
- Intended public product blocked by a new pattern: Narrow or remove the new restriction before release.
- Intentionally excluded URL in your intended indexable set: Reconcile the page policy with that set rather than assuming a sitemap will override it.
Follow the actual Sitemap: destinations in the proposed output. Check that they resolve to the intended sitemap content and that you have not dropped a useful declaration while rebuilding the template. Use absolute URLs.
A robots Allow rule does not make a page indexable by itself, and a sitemap does not guarantee inclusion. Keep the records separate: may be fetched, intended to be indexed, and currently observed in search.
For the underlying mechanics, see how to use robots.txt without hurting SEO. Your approval record should show the storefront-specific decisions, not just a generic validator's green result.
10. Distinguish open-web AI access from Catalog distribution
If the proposed change concerns AI crawlers, identify which control you actually need. Blocking a bot's open-web fetches and changing a connected shopping channel's product distribution are different tasks.
Shopify's guidance on AI crawler access explains that product data for activated agentic storefronts can be syndicated through Shopify Catalog independently of /robots.txt. Blocking a crawler there does not stop that Catalog distribution.
| Intended decision | Where to investigate |
|---|---|
| Set an advisory rule for a named open-web crawler | The exact user-agent token and its documented purpose, then the rendered robots policy |
| Change what an activated agentic storefront receives | Shopify's agentic storefront settings and Catalog-sharing controls |
| Investigate access despite a robots block | The requester and access evidence; robots rules are advisory |
Do not paste an undifferentiated “block AI” list into the theme. Establish the role of each token and the consequence of blocking it using current first-party documentation. Keep this separate from any rule affecting Google's product or page crawling.
A robots permission is not proof that a bot fetched the page, and a block is not proof that all copies or distribution channels disappeared. State the outcome you can control, then verify the corresponding setting or response.
11. Write the acceptance test and rollback first
Before saving a published-theme change, prepare the exact before/after comparison and recovery file. Shopify treats robots.txt editing as an unsupported customization, so assign someone with the relevant code and SEO knowledge to the change.
Copy this CSV as a starting record. All URLs and campaign names are examples; replace them with actual storefront paths. Keep expected crawl permission separate from intended index policy.
host,crawler,path,expected_robots,index_policy,reason
example.com,Googlebot,/products/sample-product,allow,index,active product control
example.com,Googlebot,/collections/sample-collection,allow,index,collection control
example.com,Googlebot,/collections/sample-collection?page=2,allow,review,pagination discovery control
example.com,Googlebot,/collections/retired-campaign,disallow,review,illustrative targeted crawl exclusion
example.com,Googlebot,/collections/retired-campaign-new,allow,index,similar handle must stay accessible
Add your actual resource URLs, relevant market paths and other crawler groups. For every new rule, include both an intended match and a protected counterexample. Do not copy the example's exclusion without deciding how that page should be treated.
Save the previous template outside the theme and record the theme, timestamp, intended diff and recovery owner. Choose the recovery method in advance:
- Restore the previous custom policy: put back the reviewed template and verify its rendered response.
- Return to Shopify's managed policy: after preserving customizations, delete
robots.txt.liquidonly if the resulting managed policy is acceptable. This is a reset, not an exact restoration of previous custom rules.
After saving, capture the public file again using check 2's command with robots-after filenames. Confirm the expected response, complete diff, separate directive lines, sitemap destinations and every URL/agent result in your record. Unexpected blocks on protected URLs or unexplained unrelated rule changes are reasons to restore the approved policy and investigate.
The live file and Google's cached copy are separate evidence. Google generally caches robots.txt for up to 24 hours, and sometimes longer when refreshing fails. An immediate public change does not prove that Google has refetched it or updated page indexing.
Proceed only when the full change has a clear purpose, its protected URLs pass and its recovery path is ready. If the managed file already meets the need, keeping it is a completed decision.