Free Robots.txt Tester and Validator
Check whether a URL is allowed or blocked for Google, AI search, training, and custom crawlers. Test a live website or paste a robots.txt file to see the exact rule and line that applies.
HOW IT WORKS
Test a crawl rule in three steps.
Check one URL without creating a project or account. Live mode reads only the website's public robots.txt file.
Choose the source
Fetch the public robots.txt file from a website or paste content that you want to test privately in your browser.
Select a crawler
Choose a search, AI search, training, user-triggered, or custom crawler product token and enter the URL path to test.
Read the exact match
See whether the URL is allowed, which group applies, and the exact Allow or Disallow rule and line behind the result.
MATCHING RULES
Understand why a crawler is allowed or blocked.
A useful result should show the rule behind the decision, not only a green or red status.
Most specific rule wins
The tester applies longest-match behavior. When equally specific Allow and Disallow rules conflict, the less restrictive Allow rule wins.
Wildcard and end matching
Use an asterisk to match a sequence of characters and a dollar sign to anchor a rule to the end of the tested URL.
User-agent groups matter
A crawler-specific group takes priority over the global * group. Different search, training, and user-triggered agents should be tested separately.
No rule means default access
When no applicable Allow or Disallow rule matches, the URL is allowed by default under the robots exclusion protocol.
CRAWLING VS INDEXING
Robots.txt controls access, not search inclusion.
Crawl control
Allow and Disallow rules tell a compatible crawler which URL paths it may request.
Index control
Use crawlable noindex directives and search-engine removal workflows when a URL should not appear in search results.
Robots.txt tester questions.
Clear answers about crawl rules, AI crawler tokens, error states, and indexing.
What does a robots.txt tester do?
A robots.txt tester compares a crawler user-agent and URL against the rules in a robots.txt file. It shows whether crawling is allowed, which user-agent group applies, and the exact matching rule when one exists.
Does robots.txt prevent a page from being indexed?
Not by itself. Robots.txt controls crawling, not indexing. A blocked URL can still be discovered through links and may appear without a full snippet. Use an appropriate noindex directive on a crawlable page when removal from search results is the goal.
What does “Allowed by default” mean?
It means the tester found no applicable rule that blocks the URL. This can happen when there is no matching user-agent group, the matching group has no relevant rule, or no usable robots.txt file exists.
Can I test AI crawlers separately from Googlebot?
Yes. Search crawlers, AI search crawlers, model-training crawlers, and user-triggered fetchers use different product tokens and can have different rules. Select each agent separately to check the policy that applies to it.
What do * and $ mean in robots.txt?
An asterisk matches a sequence of characters in a URL path. A dollar sign anchors a pattern to the end of the URL. The tester supports both patterns when identifying the most specific applicable rule.
What happens when robots.txt returns a 404 or 403?
Major crawlers generally treat most 4xx responses as if no crawl restrictions were published. A 429 response is different because it signals temporary rate limiting, so the tester reports an undetermined live state.
Why is a 5xx or timeout result undetermined?
A temporary server or network failure does not prove that a crawler is allowed or blocked. Crawlers may pause requests or reuse a previously cached robots.txt file, so the tester avoids presenting a definitive rule decision.
Does Screpy store the URLs or pasted robots.txt content?
No. Pasted content is analyzed in your browser and is not uploaded. Live mode sends only the website, tested URL, and selected crawler for the current check, and the tool does not store those inputs.