The Robots.txt Tester checks whether a website's rules allow a selected crawler to request a specific URL. Test the live file or paste a draft, then inspect the matching group, directive and line. The result explains crawl permission under those rules; it does not confirm that a bot visited the page or that the URL is indexed.
Choose the robots.txt source and URL
Use Live site to read the public file currently served by your website. Enter https://example.com, the path /products/item, and the crawler you want to check. Screpy requests /robots.txt; it does not fetch the tested page or crawl the whole site.
Use Paste robots.txt for an unpublished draft. The pasted text is analyzed in your browser without uploading it. Supply the website address as context so the tester can resolve the target URL.
Keep the scheme and hostname exact: https://example.com, http://example.com and https://www.example.com are different origins. A bare domain uses HTTPS. A full test URL must have the same origin as the website input; a path is resolved against it. Testing / checks only the homepage path. Preserve the actual capitalization and query string when checking another URL.
Choose a crawler preset or enter a custom product token such as ExampleBot, rather than a full browser user-agent header.
Read the robots.txt test result
Read the status alongside its evidence, especially the matching group and highlighted line.
| Status | Meaning | What to inspect |
|---|---|---|
| Allowed — green | A matching Allow rule permits this URL. |
The winning directive and line. |
| Blocked — red | A matching Disallow rule blocks this URL. |
Whether the path and crawler are the intended ones. |
| Allowed by default — blue | No applicable path rule restricts this URL. | The selected group and whether a usable live file was returned. |
| Undetermined — yellow | The live check cannot establish a rule decision. | The source response or retrieval warning. |
For a live check, inspect the robots.txt URL and HTTP status first. This tester treats 404, 403 and other 4xx responses except 429 as having no published restrictions. A rate limit, server failure or unresolved redirect leaves the decision undetermined; a timeout can show an error instead of a result. These outcomes do not prove an explicit Disallow. Real crawlers may have a cached file or different failure handling; Google documents its own response rules.
Review parsing warnings before trusting an unexpected default result. A misspelled directive or missing colon can leave the intended rule unused. Screpy processes at most the first 500 KiB and flags truncation. When content is truncated, the decision describes the processed portion; later rules have not been checked.
Understand which robots.txt rule wins
A crawler-specific group takes priority over User-agent: *; the two are not inherited together. Within the applicable group, the longest matching path rule wins. If equally specific Allow and Disallow rules match, Allow wins. Matching groups for the same specific agent can combine, so inspect repeated groups too. These principles are described in the Robots Exclusion Protocol.
Paste this illustrative policy and compare the following tests:
User-agent: *
Disallow: /private/
Allow: /private/help/
User-agent: Googlebot
Disallow: /preview/
| Crawler | Tested path | Expected result | Reason |
|---|---|---|---|
| Bingbot | /private/report |
Blocked | The global /private/ rule matches. |
| Bingbot | /private/help/guide |
Allowed | The longer /private/help/ exception wins. |
| Googlebot | /private/report |
Allowed by default | Its specific group has no matching rule; the global block is not inherited. |
| Googlebot | /preview/page |
Blocked | Its specific /preview/ rule matches. |
The example demonstrates matching, not a secure private area. Paths are case-sensitive: /Private/ is different from /private/. In path patterns, * matches a sequence of characters and $ anchors the end. For example, Disallow: /*.pdf$ matches /guide.pdf, but not /guide.pdf?download=1. Test the exact URL you use rather than assuming nearby variations share its result.
Test search and AI crawlers separately
The selected product token changes which rules apply. Check Googlebot and Bingbot individually when search crawling is the concern. For AI agents, distinguish search, model training and user-requested fetching instead of assuming one rule covers them all.
For example, OAI-SearchBot serves ChatGPT search, while GPTBot crawls content that may be used for model training. ChatGPT-User handles certain user-initiated actions, and robots.txt rules may not apply to those requests, according to OpenAI's crawler documentation.
Selecting a preset evaluates that token against the file. The live check does not impersonate the chosen bot, authenticate its requests or prove that it follows the displayed policy. A blocked result for a user-triggered token is a rule interpretation, not a guarantee that requests will stop.
Fix a rule and retest the live file
Change the smallest relevant rule, then check both the intended restriction and the exception you want to preserve.
- Copy the current file into paste mode and reproduce the unexpected result with the same URL and crawler.
- Correct the matching directive or group. Test the affected path and a neighboring URL that should remain allowed or blocked.
- Repeat for every relevant crawler. A successful Googlebot check does not establish the result for Bingbot or GPTBot.
- After publishing the file on your own website, switch to live mode and repeat the checks. Confirm the returned file and matching line reflect the change. A fresh tool response does not prove that a real crawler has refreshed its cached copy.
If the goal is removal from search, Disallow alone is insufficient. Google must be able to crawl a page to read its noindex directive. Protect confidential content with authentication, not robots.txt. Google explains the limits of crawl blocking. Use Noindex Checker to inspect the page’s HTML and HTTP directives, including their source and applicability to Google web search.
The report also lists Sitemap: declarations; finding an address does not validate its XML or prove indexing. Open it in the Sitemap Checker for that separate check. When crawler access is correct but a page still needs investigation, use the On-Page SEO Checker to review its page-level SEO signals. For AI crawler rules, compare policies with AI Bot Access Tester. This checks robots.txt only and does not establish actual bot access, indexing or citations.