Robots.txt Testers for AI Search Crawlers

Robots.txt testers for AI search crawlers are useful only when they let you evaluate the crawler you actually care about. A Googlebot result cannot tell you whether your rules allow OAI-SearchBot or Claude-SearchBot. Choose a tester that accepts the exact crawler product token, shows the applicable group and deciding rule, and identifies which robots.txt file it evaluated.

That result answers a permission question. It does not prove that the real bot can fetch your landing page through a firewall, read its content, or include it in an answer.

For a single URL, a browser-based rule tester is often enough. For several templates and agents, a batch matrix or crawl-based workflow is easier to review. The options below are compared against their published documentation, followed by an illustrative policy you can use to check your own results. This is not a hands-on benchmark of their parser accuracy or live AI crawler behavior.

Choose the AI crawler token before choosing a tester

Start with the policy decision: which service should be able to use which public pages? “Allow AI” is too broad to test. Search retrieval, model training, and a page visit initiated by a person can have different controls, even when the same company operates them.

OpenAI documents independent settings for OAI-SearchBot and GPTBot. A site can allow search crawling while opting out of training collection. Its user-initiated ChatGPT-User agent is different again: OpenAI says robots.txt rules may not apply to those requests, and this agent does not determine Search eligibility. Do not use a simulated ChatGPT-User rule result as proof that a user's request will be blocked.

Anthropic identifies three separate agents and says its bots honor robots.txt directives. Test Claude-SearchBot, ClaudeBot, and Claude-User independently when your policy covers all three. An exception documented by one provider should not be assumed for another.

The Google crawler reference explains that Google-Extended has no separate HTTP request user-agent. It is a robots.txt control token for Gemini training and grounding, and does not affect Google Search inclusion or ranking. Blocking it is not a general switch for excluding a website from Google's AI search features.

Enter the product token, such as OAI-SearchBot, when a tester asks for that identifier. A full browser-like HTTP header is a different input. If a tool accepts both, check how it resolves the header to a matching group. The right input is the one its documented parser expects, not the longest string you can copy from a bot directory.

Robots.txt testers that accept AI crawler tokens

A useful shortlist needs explicit crawler selection, a reproducible rule source, and an explanation of the result. An “AI ready” badge on a homepage is not a substitute for those capabilities.

Screpy publishes this guide and offers one of the tools listed. The comparison is based on public documentation checked on October 11, 2026. It does not establish which parser is most accurate, fastest, or best for every site.

For Screaming Frog, pay attention to the two user-agent settings. Robots User-Agent determines which robots directives it follows. HTTP Request User-Agent changes the request header. Setting the latter alone is not an adequate description of your rule test. Its custom robots feature overrides the file for that crawl; it does not edit the live website's robots.txt.

For an online tester, check what is sent to the service before pasting an unpublished policy. RobotSense documents backend parsing; other tools describe browser-side parsing or a server used to fetch the public file. Do not assume all editors keep pasted content local. Avoid putting credentials or confidential URLs in any public tool.

MySmartCoPilot documents fetching robots.txt as its own bot. That matters if your server returns different file content by client identity. A tool's live fetch can be accurate for its request while differing from the file an actual AI crawler receives.

Use the provider documentation for crawler purpose and search behavior, regardless of the tester's wording. A helpful tool can still have an oversimplified explanation. Custom-token support makes a tester adaptable; it does not certify that its implementation matches every provider-specific exception.

What a useful AI crawler test result must show

Save enough information for someone else to reproduce the decision. A screenshot saying “Allowed” is weak evidence when the file, origin, token, or path is missing.

Your test record should contain:

• Rule source: live fetch or pasted candidate, plus a saved copy or version identifier.
• Origin: protocol, hostname and relevant port, not just a brand or domain label.
• Crawler input: the exact product token and any parser profile selected.
• Target: the full URL and path being evaluated, preserving case and relevant query strings.
• Decision: allowed, blocked, or undetermined, with a reason.
• Matching evidence: applicable group, deciding directive, and line where available.
• Fetch context: robots.txt response status, final file URL and retrieval time for a live check.

For example, evaluating /guides/start/ against a pasted candidate is useful before a change. It does not establish that https://example.com/robots.txt currently serves that candidate. Label those two checks separately in the ticket.

Likewise, test the origin used by your intended public URL. An HTTPS file on the main host is not evidence about a different subdomain's policy. If a redirect leads to another host, investigate the destination's rules as a separate part of the access path rather than treating one verdict as a site-wide answer.

Ask why the result is allowed. It may be a matching Allow directive, no applicable restriction, or a file-fetch state that the tester interprets as unrestricted. Those outcomes need different explanations. If robots.txt retrieval fails, preserve the error instead of copying a green badge without context.

The general Screpy robots.txt guide covers file location, directives, and crawl-versus-index control in more detail. Here, the acceptance criterion is narrower: can another person reproduce this URL's permission decision for this agent from the same input? If not, your evidence is incomplete even when the verdict happens to be correct.

Test search and training rules with the same URL set

Use a small, understandable fixture before trusting a large matrix. The following file is an illustrative candidate, not a replacement for your production policy. Its purpose is to check group selection and path precedence for a few exact tokens.

User-agent: *
Disallow: /private/

User-agent: OAI-SearchBot
Disallow: /private/
Allow: /private/public-guide/

User-agent: GPTBot
Disallow: /

The intended policy is to allow ordinary public pages, keep the private path restricted for the tested search crawler, allow one deliberately public guide within that path, and decline GPTBot collection. Other named groups and provider policies would need their own decisions.

Under the matching rules in the Robots Exclusion Protocol specification, the expected evaluations are:

These are expected decisions derived from the file, not measured results from the listed tools. Enter the same content, tokens and paths into your chosen tester and retain its actual output alongside the fixture.

Notice that the public-guide exception applies only to OAI-SearchBot. Googlebot does not inherit the named group's Allow rule. Conversely, OAI-SearchBot does not inherit the wildcard group's restrictions, which is why the private restriction is repeated in its own group.

As a negative check, remove that repeated Disallow: /private/ from the OAI-SearchBot group in a separate pasted copy. The private-account path then has no blocking rule in that group and should evaluate as allowed. That deliberate difference helps reveal a parser or input mistake that a homepage-only check would miss.

Add a second small fixture with equally specific Allow and Disallow values when you need to check tie handling. Keep complex wildcard and encoded-path cases separate so you can identify which behavior differs. One passing example does not establish complete standards compliance.

Finally, a directory called private is not secured by these instructions. Protect genuinely confidential content with authentication and appropriate access controls. Preserve existing search, asset and platform rules when making a real change; this fixture intentionally omits them to make the test legible.

When robots.txt says allowed but the crawler still fails

Treat an allowed result as one passed layer, then investigate the layer that actually failed. Otherwise you can spend hours editing a rule that was never the problem.

Separate the robots.txt response from the landing-page response in your notes. A 403 from the page and a 403 from robots.txt are different events. The Google robots.txt specification describes Google's own error handling and caching. Do not turn those rules into a universal prediction about every AI provider, particularly when the tester reports a timeout or an undetermined state.

A request sent from your laptop with a copied bot header is still your request. It does not originate from the provider's network or prove that a firewall recognizes the genuine crawler. When diagnosing actual access, use provider verification guidance and the relevant request logs; record how identity was verified rather than trusting the header alone.

If Cloudflare is the enforcement layer, use the guide to allowing AI search crawlers without allowing AI training for the configuration workflow. A rule tester can explain your policy, but cannot change or verify every Cloudflare control on your behalf.

After a correction, refetch the file and repeat the same URL/token pairs. Retain the before and after evidence. A fresh tester result shows what its new input implies; it does not establish when another crawler replaces a cached copy or revisits the page. Confirm those events through actual observations when they matter to the diagnosis.

Choose a tester by the decision you need to make

Choose a simple rule explainer when you need to answer one question in a support ticket. Choose a batch-capable tool when several templates must behave differently across search and training tokens. Choose a crawl-based test when you already have a URL inventory and want to assess the reach of a proposed policy across it.

Before relying on any of these robots.txt testers for AI search crawlers, give it a small acceptance check:

1. Load a file whose intended behavior you understand. Keep live and candidate versions distinct.
2. Evaluate a normal public page, a restricted path and a deliberate exception for each relevant token.
3. Include a fallback case and a negative case that changes when you remove a rule.
4. Inspect the applicable group and winning directive rather than accepting the verdict alone.
5. Preserve the inputs and actual results, then investigate any disagreement with the expected policy.

Expand your URL set around templates that matter: product pages, documentation, articles, account routes, language variants and assets required to understand those pages. Select representative paths deliberately. A hundred copies of the same pattern can be less informative than six paths covering different restrictions.

If two testers disagree, first compare inputs. Did one fetch a different origin, use a different token, load a cached file, or select a provider-specific parser profile? If those are identical, reduce the file to the smallest conflicting example and compare it with the relevant standard and provider documentation. Do not settle the disagreement by counting green badges.

Record a successful rule check as “this file permits this path for this token under this evaluator.” Record actual bot access separately when you have verified request evidence. That precise language makes the report useful to an SEO specialist, a developer and the person responsible for access policy.

Repeat the accepted fixture after meaningful robots.txt, domain, redirect or bot-policy changes. Keep the expected decisions under version control or in your audit record. The value of the tester is a decision you can explain and reproduce, not a promise that an AI service will recommend your website.