If you want AI search crawlers to reach your site without giving every training crawler the same access, use Cloudflare's new purpose-based controls. Set Search to Allow and Training to Disallow AI Training. Avoid setting Training to Block when continued access for mixed-purpose search crawlers matters, because Cloudflare now applies that stronger action to crawlers that serve both search and training functions.
That is the configuration. The rest of the job is proving it works. You need to inspect the public robots.txt, confirm that Cloudflare is not returning a challenge or 403 to the crawlers you intend to admit, and check actual crawler response data after the change. A permissive robots rule cannot override a stricter WAF or bot policy.
This guide explains the controls Cloudflare introduced on September 15, 2026, shows a safe baseline for SEO and AI discovery, and provides a verification workflow. It does not promise indexing or citations. Crawl access only makes discovery possible; it does not determine whether a search or AI system will use your page.
Crawler access is the technical eligibility layer of AI visibility in SEO. It can make retrieval possible, but it does not guarantee a mention, citation, recommendation, or ranking.
Use the New Cloudflare Controls, Not the Old All-or-Nothing Toggle
Older advice often tells site owners to switch “Block AI Bots” on or off. That advice is now incomplete. On September 15, 2026, Cloudflare moved to three behavior-based policies: Search, Agent, and Training. The company is also deprecating the legacy Block AI Bots control in favor of these separate choices.
The distinction matters because a crawler name does not always describe one purpose. Cloudflare classifies some crawlers, including Googlebot, Bingbot, and Applebot, as mixed-purpose: the same crawler infrastructure may support search and training-related uses. Under the current policy, choosing Block or Block on pages with ads for Training can block those mixed-purpose crawlers. That can interfere with the search access you intended to preserve.
Use the policy purpose as your starting point:
| Cloudflare policy | What it covers | Baseline for a public, search-dependent site |
|---|---|---|
| Search | Crawlers that collect or index content so it can be found later | Allow |
| Agent | Automated activity performed in real time for a person, such as chat fetches or browser agents | Decide from your product and security needs |
| Training | Crawlers collecting content for model training or fine-tuning, including mixed-purpose crawlers | Disallow AI Training |
The wording of the Training action is crucial. Disallow AI Training publishes a no-training preference while continuing to admit mixed-purpose crawlers that Cloudflare designates as accountable. Block denies the crawler request at the edge, so it can also remove its search function. Cloudflare documents the current behavior in its AI bot policy reference and September 2026 announcement.
This is a safer default, not a universal policy. A private portal, licensed archive, or site with no discovery goal may make a different choice. The important change is to decide by use instead of treating every AI-related request as equivalent.
Choose a Policy by Purpose Before Touching Bot Names
Write down the outcome you want before changing a crawler rule. “Allow AI bots” is too broad because search indexing, real-time retrieval, and model training are different activities.
For most public marketing sites, the practical goal is:
- Keep traditional search engines able to crawl indexable pages.
- Let selected AI search products discover or retrieve public content when that supports the site's visibility strategy.
- Refuse use for model training unless the business has chosen to allow it.
- Keep account areas, checkout, internal search results, staging sites, and private data protected for every crawler.
Provider documentation shows why purpose matters. Google says Google-Extended is a standalone robots token for controlling specified Gemini training and grounding uses; blocking it does not affect inclusion or ranking in Google Search. Googlebot remains the crawler that controls access for Google Search, including its AI features. Anthropic documents three separate identities: ClaudeBot for content that may contribute to model training, Claude-SearchBot for search, and Claude-User for retrieval initiated by a user.
These examples are useful for understanding the model, but they should not become a copied-and-forgotten master list. Crawler identities and product purposes change. Cloudflare's current classifications and each operator's current documentation should be checked when you make the decision.
Also separate access from content strategy. Allowing a crawler to request a page does not make that page useful, authoritative, or likely to be cited. If the broader goal is discovery in answer engines, review the content and measurement steps in Screpy's guide to improving AI visibility after the access layer is correct.
Configure Search as Allow and Training as Disallow AI Training
Cloudflare documents the current controls under Security Settings → Configure AI bot policies. The interface may move as the product evolves, so use the setting names rather than relying only on a screenshot.
For a public site that depends on search discovery:
- Open the correct Cloudflare account and zone.
- Go to Security Settings.
- Filter or navigate to the AI bot policies.
- Set Search to Allow.
- Set Training to Disallow AI Training.
- Choose the Agent policy separately based on whether you want real-time, user-directed tools to access public pages.
- Save the policy, then continue to the verification steps below.
Here is what the Training options mean in practice:
| Training action | Effect | Search risk |
|---|---|---|
| Allow | Cloudflare adds no training block through this policy | Training access remains possible unless another rule restricts it |
| Disallow AI Training | Publishes the applicable no-training preference and allows accountable mixed-purpose crawlers to continue their search role | Designed to preserve search while refusing training use |
| Block on pages with ads | Blocks training-classified crawlers, including mixed-purpose crawlers, on pages Cloudflare detects as showing ads | Can block search crawling on those pages |
| Block | Blocks training-classified crawlers, including mixed-purpose crawlers, across the zone | Can block search crawling across the site |
Do not treat Agent as another word for Search. Agent traffic can include a browser or fetch tool acting for a person in real time. A documentation site may want to allow it; an application with expensive endpoints or sensitive workflows may choose a stricter policy. Make that decision by hostname and content type where your Cloudflare plan and rules support it.
Finally, review independent controls. An Allow choice here means this policy does not add a block. It does not cancel a WAF custom rule, rate limit, challenge, access policy, origin firewall, or application login. If a search crawler still receives a 403, the saved AI policy is only the first place to check.
Verify robots.txt and Cloudflare Enforcement Separately
A correct dashboard selection is not enough. Verify the policy at each layer, because robots.txt and edge enforcement solve different problems.
1. Read the public robots.txt response
Open https://yourdomain.com/robots.txt as an unauthenticated visitor. Confirm that it returns a readable text response rather than a login page, challenge, redirect loop, 403, or 404. If Cloudflare manages the file, look for a search-allowed and training-disallowed preference. Current managed output may include a line similar to:
User-agent: *
Content-signal: search=yes, ai-train=no, use=reference
Allow: /
Treat this as an example to inspect, not text that every site should paste over its existing file. Preserve rules for private paths, platform resources, and sitemaps. Cloudflare may also publish crawler-specific groups.
Cloudflare is explicit that robots.txt compliance is voluntary. The file communicates a preference; it does not technically prevent a request. AI Crawl Control or another edge rule provides enforcement for crawlers that do not comply.
You can use Screpy's robots.txt tester to test a live file or pasted draft against individual search, AI search, training, and custom crawler tokens. Test the exact URL paths that matter, not only /.
2. Check representative resources
Verify more than the homepage. Use a public article or product page, robots.txt, and the sitemap because a path-specific rule can make one resource accessible while blocking another.
| Resource | Expected result for an allowed search crawler |
|---|---|
/robots.txt |
Readable policy response |
| Public landing page | Successful response with the intended content, without a challenge |
| Representative deep page | Same access outcome as the page type you want discovered |
| XML sitemap | Successful response if the crawler uses it for discovery |
3. Inspect Cloudflare's crawler evidence
In AI Crawl Control, review crawler activity and filter by crawler, operator, hostname, path, status code, and time range. The Directives view can show robots availability and apparent violations. Cloudflare notes that violation reporting can compare current directives with historical requests, so a request made before a rule change may appear as a violation later.
If the crawler receives 403, 429, or a challenge page, inspect Cloudflare Security Events and the matching rule. Look for a custom WAF rule, rate limit, Bot Fight setting, access policy, or origin response that conflicts with the AI policy. A cached robots.txt can also leave an old rule live after the dashboard changes.
Screpy's AI & Bot Traffic view can help after a Cloudflare zone is connected. It reports verified crawler activity, successful and non-2xx responses, transferred bytes, trends, and affected URLs from the connected data. It does not change Cloudflare settings, and a crawler request does not prove that an AI answer cited the page.
4. Do not authenticate a bot by name alone
Changing a request's User-Agent header is useful for checking how your own rules branch, but it does not prove what a verified crawler receives. Anyone can send Googlebot or OAI-SearchBot as a string. Use Cloudflare's verified-bot evidence, supported detection fields, and operator verification methods when identity matters. Cloudflare says free-plan AI Crawl Control detection is based on known user-agent strings, while enterprise Bot Management can use detection IDs, so interpret the evidence within your plan's capabilities.
Test Search, Training, and User-Triggered Crawlers as Separate Products
After the Cloudflare policy is correct, check the product identities that matter to your site. Do not assume that allowing one crawler from a company allows every product from that company.
| Operator | Search or discovery | Training control | User-triggered access |
|---|---|---|---|
Googlebot controls access for Google Search, including its AI features |
Google-Extended controls specified Gemini training and grounding uses without affecting Google Search |
Product-specific agents may have separate controls | |
| OpenAI | OAI-SearchBot supports discovery for ChatGPT search |
GPTBot is the token publishers can disallow to exclude pages from potential training |
User-directed and product-specific crawlers should be evaluated separately |
| Anthropic | Claude-SearchBot supports search quality |
ClaudeBot may collect content that contributes to model training |
Claude-User retrieves content in response to a user's request |
This table explains purpose separation; it is not a permanent allowlist. Review the provider's current documentation before deployment. OpenAI currently tells publishers who want content eligible for ChatGPT summaries and snippets not to block OAI-SearchBot, while sites that want to opt out of potential training should disallow GPTBot. Anthropic separately documents training, search, and user-directed agents. Google says Google-Extended is a control token rather than a distinct HTTP User-Agent, so you should not expect to find “Google-Extended” as a request string in server logs.
Run four checks for each product you care about:
- Does the applicable group in the live
robots.txtexpress the intended rule? - Does Cloudflare's configured category match the purpose you intend to allow or restrict?
- Do verified requests receive the expected response on representative public paths?
- Does any broader WAF, rate limit, challenge, or origin rule override that result?
Be careful with broad User-agent: * rules. A later or more specific group may change which directive applies, and different crawlers implement the Robots Exclusion Protocol within their documented behavior. Test each token against the final public file.
Avoid allowlisting a large, static collection of IP addresses copied from a third-party post. Use the operator's current verification method or Cloudflare's verified-bot evidence. IP ranges and delivery infrastructure can change, and a spoofed User-Agent is not proof of identity.
Measure Discovery Without Treating Crawler Access as a Citation
The technical success condition is narrow: the search crawlers you chose to allow can retrieve the intended public resources, while the training preference or block you selected is present and enforced as designed. That result does not prove indexing, ranking, an AI mention, or referral traffic.
Use an evidence ladder after the change:
- Policy evidence: The Cloudflare settings and live
robots.txtexpress the intended Search and Training choices. - Request evidence: Verified crawler requests receive successful responses on the relevant paths, while blocked activity receives the intended outcome.
- Discovery evidence: Search tools and operator reports show that important pages can be found and processed. Allow time for recrawling.
- Answer evidence: A controlled set of AI prompts shows whether the brand or page appears, with the cited URLs recorded.
- Business evidence: Analytics distinguishes crawler requests from real referral sessions and conversions.
Screpy separates two useful parts of this process. AI & Bot Traffic uses connected Cloudflare data to help review verified crawler activity, response outcomes, and non-2xx URLs. AI Visibility monitors selected prompts for mentions, answer position, sentiment, citations, sources, and competitor share of voice. A bot request belongs to the first dataset; an observed citation belongs to the second. Neither one alone proves a lasting search result or a causal SEO improvement.
Record the date of the policy change and compare evidence from before and after it. If allowed crawlers begin receiving 403 or 429 responses, inspect the matching Cloudflare event and roll back the conflicting rule rather than disabling every security control. If requests succeed but visibility does not change, investigate content relevance, indexability, internal linking, and source authority instead of repeatedly loosening crawler access.
Recheck the setup when Cloudflare changes its bot categories, when a provider changes crawler documentation, or when a WAF policy is deployed. This configuration is a maintained access decision, not a one-time SEO switch.
Frequently Asked Questions
Does Disallow AI Training block Google Search?
Cloudflare designed Disallow AI Training to preserve search access for accountable mixed-purpose crawlers while publishing a no-training preference. Google also states that restricting Google-Extended does not affect inclusion or ranking in Google Search. A separate WAF or bot rule can still block Googlebot, so verify actual responses after the change.
Is robots.txt enough to stop AI training crawlers?
No. robots.txt communicates a preference to compliant crawlers; it is not an authentication or access-control mechanism. Use Cloudflare's AI crawler controls or appropriate WAF rules when you need technical enforcement, then monitor whether blocked crawlers continue requesting disallowed paths.
Should I allow Cloudflare's Agent category?
It depends on the site. Allowing Agent traffic can support user-directed retrieval and browser tools on public content. Blocking or limiting it may be appropriate for account areas, expensive endpoints, transactional flows, or pages where automated interaction creates risk. Decide separately from Search and Training.
How quickly will the change affect search or AI visibility?
The edge policy can change once Cloudflare deploys the saved configuration, but external search and AI systems must revisit and process the pages. That can take longer and varies by product. Verify the live policy first, then watch crawler responses and downstream discovery over time rather than expecting an immediate citation.