AI search bot rules
Enter a domain or full HTTP/HTTPS URL. The path and query are preserved; robots.txt is fetched from its origin. A matching specific group takes priority over the wildcard group. Longer matching rules take priority; Allow wins at equal specificity. Up to five redirects are followed.
AI search
OAI-SearchBot · Claude-SearchBot · PerplexityBot
If AI search discovery is intended, review the matching Disallow rule before changing it. This checks robots.txt policy only. It does not prove actual bot access, indexing or citations.
Model training
GPTBot · ClaudeBot
Blocking a training bot can be intentional. It is not automatically an SEO problem.
User-requested visits
ChatGPT-User: Robots.txt may not apply to user-requested visits.
Claude-User respects robots.txt for user-requested visits.
Perplexity-User: This bot generally ignores robots.txt for user-requested visits.
Content use controls
Google-Extended: This controls content use, not a separate HTTP crawler or Google Search.
Source
An inaccessible response, invalid document or exceeded limit produces an unknown result. A missing file returning 404 or 410 means no robots.txt restriction was found. The robots.txt response is limited to 512 KiB.
Noindex Checker · robots.txt · SEO
OpenAI · Anthropic · Perplexity · Google-Extended
Test the URL you actually want discovered
Check a public article or product URL, not just the homepage. A rule allowing / can coexist with a restriction on /private/; the answer depends on the requested path. For https://example.com/private/report/, inspect the applied group and matching rule before deciding whether access is intentional.
The report separates allowed, blocked and unknown policy results. Read the search-bot rows first if your goal is discovery. Do not interpret a green total as permission for every bot category or proof that the content will appear in an answer.
Keep search and training decisions separate
An owner can allow OAI-SearchBot while blocking GPTBot. Those choices serve different purposes: making pages available for search does not require treating every training crawler identically. The same distinction matters when reviewing other named bots. Use their published documentation to confirm the purpose of each user agent.
Recheck after changing robots.txt
Edit the rule at the origin where the tested URL lives, publish it, then rerun the same exact URL. Check a second URL outside the restricted directory to catch an overly broad change. Retain an intentional restriction rather than removing every Disallow line.
If policy is allowed but access still fails, investigate authentication, HTTP responses and server restrictions separately. This checker cannot verify those controls. Use Bulk HTTP Status Checker for response evidence and Noindex Checker for indexing directives.