Screpy’s AI Bot Access Tester checks the robots.txt policy for an exact URL against nine named bots or content-use controls. It separates AI search, model training and user-requested visits so you can review the rule that matches your goal. The result shows what the published policy says, not whether a real bot reached the page, indexed it or cited it in an answer.
Test the exact URL and its origin
Enter a domain or a full public HTTP or HTTPS URL. A domain alone checks its root path; a full address preserves the path and query when evaluating rules. If your concern is a product guide, test that guide rather than assuming the homepage result applies everywhere.
The checker retrieves robots.txt from the supplied URL’s origin. Different subdomains and HTTP/HTTPS origins can have different files, so check the address you actually publish. The source details show the requested robots.txt address, its final address after any followed redirects and the check time. Up to five source redirects are followed.
Use Subdomain Finder to find other hostnames to review, then check each relevant origin's bot policy separately. A certificate record does not confirm a working website or actual bot access.
The page URL is the target for rule matching, not a page whose content is analyzed here. A successful robots.txt retrieval therefore does not establish that the target article is reachable or readable. Keep the tested address with your result when comparing policy before and after a change.
Separate AI search, training and user-requested visits
A blocked training crawler is not automatically a search-discovery problem. Read the purpose of each row before deciding that it should be green. The current report covers these nine entries:
| Purpose | Entries in the report | How to interpret the category |
|---|---|---|
| AI search | OAI-SearchBot, Claude-SearchBot, PerplexityBot | Review these rows when the goal is discovery in the corresponding search experience |
| Model training | GPTBot, ClaudeBot | A training opt-out can be intentional and separate from search policy |
| User-requested visits | ChatGPT-User, Claude-User, Perplexity-User | Read the behavior note; these three do not treat robots.txt identically |
| Content-use control | Google-Extended | A content-use token, not a separate HTTP crawler or a Google Search ranking control |
OpenAI’s crawler documentation distinguishes OAI-SearchBot from GPTBot and says robots.txt may not apply to user-triggered ChatGPT-User requests. Allowing search while blocking training is therefore a meaningful separate policy choice, not an inconsistent result.
Anthropic’s bot guidance distinguishes Claude’s search, training and user-requested agents and states that its bots respect robots.txt. Perplexity’s crawler guidance separates PerplexityBot search from Perplexity-User, which generally ignores robots.txt for user-requested fetches.
Google’s Google-Extended documentation describes content-use controls and states that the token does not affect inclusion or ranking in Google Search. Do not treat its row as a test of Googlebot or all Google AI search features. Bot purposes and documented behavior can change; use these primary references when reviewing a long-lived policy.
Read the applied group and matching rule
The evidence identifies the bot, applied group, matching Allow or Disallow rule and source line. A matching specific bot group takes priority over the wildcard group. Within the applicable rules, longer matching paths take priority; Allow wins at equal specificity. No matching restriction means the checker found no disallow rule for that target under the applied policy. It does not mean an explicit Allow line is required.
Consider a fictional OAI-SearchBot group containing Disallow for /private/ and a more specific Allow for /private/public-guide/. The public-guide path can be allowed while /private/report/ remains blocked. Adding a separate GPTBot group that blocks all paths does not change those OAI-SearchBot decisions. This illustrates rule matching, not a recommended policy to paste into your site.
Unknown is different from blocked. An inaccessible source, invalid document or exceeded limit leaves the policy unresolved. The source is limited to 512 KiB. A missing robots.txt returning 404 or 410 means no restriction was found in that file, not that authentication or other controls disappeared. Review source warnings and directive line diagnostics before changing anything. For a custom user agent or a closer rule review, use the Robots.txt Tester.
Change only the policy you intend to change
If AI search discovery is intended, start with the search-bot rows and inspect the matching restriction. Keep intentional training or private-path restrictions instead of deleting every Disallow line. Edit the responsible origin’s file, publish the change and retest the same exact URL. Also check a URL that should remain restricted to catch an overly broad edit.
An allowed policy is not an actual-access test. The checker does not send requests as all nine agents, inspect bot traffic logs or verify authentication and server access rules. A total of allowed rows is not a visibility score, and a missing file is not a guarantee of unrestricted access.
Investigate HTTP response evidence separately with the Bulk HTTP Status Checker, remembering that its request also cannot prove what every genuine bot receives. Use the Noindex Checker for page-level indexing directives. Robots policy, successful retrieval, indexing and selection as a cited source are separate questions. Changing one rule does not guarantee traffic, recommendations or citations.
Before publishing your file, use llms.txt Validator to check its Markdown structure and discovered links. Optional HTTP checks inspect a bounded set of destinations; they do not establish that an AI agent reads the file.
For a broader source review, use AI Search Readiness Checker to inspect page HTML, robots policies and bounded discovery files together. Its technical findings do not measure AI visibility or guarantee citations.