Publishers and content libraries
See whether verified search and AI crawler families are requesting the site and whether important content paths return non-2xx responses.
AI & Bot Traffic Analytics
Connect the Cloudflare zone matched to your Screpy project and review the latest seven days of verified bot activity. Compare crawler categories, publishers, data transfer, request trends, successful responses, and non-2xx URLs without confusing a crawler visit with an AI citation.
Connect Cloudflare Bot Analytics
Add your Cloudflare connection to read verified bot requests for the zone matched to your Screpy project. Check the connection and reporting scope before interpreting an empty table.
- Use an active API token with access to the relevant zone and its analytics.
- Screpy matches the project domain to an accessible zone, including a parent zone for a subdomain.
- Traffic covers that Cloudflare zone; it can include other hostnames within the zone, not just the project's hostname.
- Missing permissions, a missing zone match or unavailable seven-day data produces an unavailable state, not proof of zero bot requests.
Source
Screpy
Screpyexample.comConnected
Identify Verified AI Crawlers
Find the search and AI crawler families present in the connected data. Names and publisher context help you distinguish who sent requests before investigating their volume or responses.
- Review recognizable families such as Googlebot, Bingbot and supported AI crawlers when their requests appear.
- The table starts with traffic classified as verified bots by Cloudflare; a crawler name alone is not verification of every automated request.
- Each crawler row shows its highest-volume category when its requests span multiple categories.
- Filter the crawler table by an available category; this selection does not filter the separate traffic chart.
Requests
42- Googlebot18
- GPTBot14
- ClaudeBot10
Compare Hourly Bot Activity
Use the category chart to spot concentrated requests, then check individual crawler sparklines for the same seven-day period. The current view has a fixed window rather than a custom date picker.
- The reporting window covers seven days of completed hourly intervals and ends at a UTC hour boundary.
- The incomplete current hour is excluded, so the chart is not a live feed of each new request.
- The five highest-volume verified categories appear separately; additional categories are combined as Other.
- A crawler's sparkline shows how its requests are distributed across the window, helping distinguish a brief burst from sustained activity.
Requests
1,860Measure Bot Requests and Data Transfer
Read request counts and transferred bytes together. They describe traffic volume and response size, not unique people, AI answer popularity or the value of your content.
- Total requests count repeated requests as well as first requests; they are not unique visitors or sessions.
- Transferred bytes sum responses served through Cloudflare, including non-2xx responses; they are not a measure of successful content reading.
- Compare each crawler's volume with its trend and outcomes before investigating a burst; these figures do not calculate hosting costs or model training.
- Requests
- 1,860
- Data transfer
- 24 MB
Understand Crawler HTTP Responses
Separate response classification from a diagnosis. The unsuccessful column includes every non-2xx response, so a high count does not by itself mean the website is broken.
- HTTP 2xx responses count as successful; all other response statuses enter the unsuccessful count.
- A 3xx redirect can be intentional. Check its destination before treating it as a crawl problem.
- A 403 may reflect an intended access restriction; the request does not prove that the crawler downloaded the page content.
- Investigate unexpected 404 or 5xx responses against the actual route and site configuration before deciding what to change.
HTTP status
200GET /HTTP/2 200Inspect Non-2xx Crawler Paths
Open a crawler's non-2xx count to see where its requests received those responses. Path, status and frequency give engineering or SEO teams a concrete investigation to follow.
- The detail view is scoped to the selected crawler and the same fixed seven-day period.
- Rows group a requested path with its HTTP status and request count; a repeated path can appear under different statuses.
- Higher request counts appear first, with pagination for the available rows. This is an aggregated view, not an individual request log.
- Record the crawler, path, status and count, check whether the response is intended, then review later activity after any verified fix.
SelectedGPTBot
| URL path | HTTP status | Requests |
|---|---|---|
/guides/seo | 404 | 128 |
/guides/seo | 301 | 36 |
/setup | 503 | 12 |
Talk about AI Bot Traffic. With SEO MCP.
Ask about Quick Wins built from available search, crawl and connected Cloudflare traffic signals through SEO MCP. Cloudflare access covers those opportunities, not the complete analytics or bot-traffic dashboard.
- Screpy: Hello! Let’s look at AI Bot Traffic for example.com. What would you like to know?
- You: Which traffic-based Quick Wins should I review for example.com?
- Screpy: Review pages with search opportunities and traffic signals in the same project. Cloudflare-based opportunities need a connected zone and available data; full analytics and bot-traffic records are not exposed through MCP. Quick Wins context: Search opportunities: Available project data. Cloudflare signals: Connection required.
- You: Turn that into a short action plan.
- Screpy: I’ll keep the evidence and recommendations separate. Review the findings, choose one change and check the results afterward. Your next steps: Review: Check the source and date of each finding.. Decide: Choose the first change to make.. Verify: Compare the results after the change..
- You: Can you explain the evidence in plain language?
- Screpy: Yes. Ask about a supported result and I can explain what it shows, where it comes from and what is still uncertain. Missing data stays unavailable; recommendations are not automatic fixes.
Turn verified bot traffic into a focused technical review.
Start with the busiest categories, select a crawler and inspect repeated non-2xx paths. Check the intended response before handing a specific route or access issue to the team responsible.
-
Connect Cloudflare and match the project zone
Use an active API token that can list the website zone and read analytics, then confirm the project domain has a matching connected zone.
-
Read the seven-day category pattern
Look for sustained activity, sudden bursts, missing expected categories, or a large Other segment before opening individual crawler rows.
-
Compare crawler volume and outcomes
Use requests, bytes transferred, hourly trend, successful 2xx responses, and non-2xx counts to choose the crawler that needs attention first.
-
Inspect paths and verify the intended response
Review each repeated path and status, confirm whether it is intentional, investigate unexpected responses, and use later traffic evidence to confirm the behavior changed.
Use crawler traffic where it changes a real decision.
The view is most useful when a team owns enough website context to decide whether a traffic pattern or HTTP response is expected.
SEO and content teams
Add real request evidence to crawlability discussions without presenting crawler activity as proof of indexing, citations, or answer visibility.
Product and engineering teams
Hand off a specific crawler, path, HTTP status, and request count when an application route or deployment needs technical investigation.
Agencies and multi-project teams
Keep each client or website zone connected to its matching Screpy project and use the same review method across separate properties.
Know what crawler traffic cannot prove.
Clear limits make the evidence safer to use in SEO, content, engineering, and AI-search discussions.
A request is not an AI citation
A verified crawler request records an access attempt and its response. Even a blocked request can appear; the record does not prove content download, indexing, training, citation or inclusion in an AI answer.
Seven days, not custom history
The current Screpy view uses a fixed seven-day period with hourly trend data. It does not promise arbitrary date ranges or long-term bot history.
Verified traffic, not every automated request
The view is built from verified bot categories available for the connected zone. It is not a raw server-log explorer or a complete inventory of every scraper.
Analytics, not crawler control
Screpy reports activity and response outcomes. The current view does not block crawlers, change access rules, send bot alerts, or export a full request log.
Choose the AI-search workflow that matches the question.
Use crawler analytics for website requests, AI Visibility for monitored answers, or Website Audit when a repeated response problem needs a broader technical investigation.
Review seven days of verified crawler categories, request trends, transfer volume, response outcomes, and non-2xx paths.
Monitor selected prompts for brand mentions, citations, source domains, answer position, sentiment, competitors, and share of voice.
Crawl the wider site when non-2xx bot requests point to redirects, broken destinations, internal-link problems, or recurring template issues.
Related SEO guides
Learn the concepts behind this workflow and apply them with practical SEO guidance.
GEO
llms.txt: What It Is, How to Create It, and Its SEO Limits
Learn how llms.txt works, how to create a useful file, what it cannot guarantee, and how to measure whether AI crawlers actually use it.
Read article GEO
How Do You Rank on ChatGPT?
Rank on ChatGPT by making pages crawlable, answering user questions clearly, and building credible sources that earn citations in relevant search responses.
Read article SEO
How to Prevent Staging Sites From Being Indexed
Staging site noindex setup: use HTTP auth, X-Robots-Tag or meta robots, plus robots.txt and Search Console checks to keep test URLs out of Google search results.
Read article SEO
How Trailing Slashes Affect SEO and Indexing
Trailing slashes SEO: avoid duplicate URLs by choosing one format and aligning 301 redirects, canonicals, sitemaps, and internal links for clean indexing.
Read articleAI & Bot Traffic Analytics FAQ
Understand the connected data source, fixed reporting period, verified crawler scope, response logic, URL drilldown, and the difference between crawler access and AI answer visibility.
Which crawlers can Screpy identify?
The table starts with traffic categorized as verified bots by the connected analytics source. Screpy adds readable identities for known crawler families such as Googlebot, Bingbot, OpenAI, Anthropic, Perplexity, Apple, Meta, and other recognized services when their requests are present. Available rows depend on actual zone traffic.
What is required before bot traffic data appears?
Connect an active Cloudflare API token that can list the relevant zone and read its analytics. Screpy then matches project domains to accessible zones. If the token lacks analytics permission, the project domain does not match a zone, or the required dataset is unavailable for that zone, the dashboard shows an unavailable state instead of substitute data.
How should I investigate unsuccessful crawler URLs?
Open the non-2xx count for the affected crawler, review each path and HTTP status, then decide whether the response is intentional. Unexpected redirects, 4xx responses, or 5xx responses can be checked against the live route, internal links, canonical destination, crawl configuration, and a wider Website Audit before a change is made.
Can Screpy block AI crawlers or change robots.txt from this page?
No. The current feature is an analytics and investigation view. Access controls, firewall rules, robots directives, crawler alerts, referral analytics, and CSV exports are not managed from this Screpy page.
What does Screpy AI & Bot Traffic measure?
Screpy uses analytics from the Cloudflare zone matched to a project to report the latest seven days of verified bot traffic. The dashboard includes an hourly category trend and a crawler table with publisher, category, bytes transferred, request totals, successful requests, non-2xx requests, and a per-crawler trend.
Is Screpy AI & Bot Traffic a server log analyzer?
No. Screpy reads available aggregated analytics for the matched Cloudflare zone; it does not ingest raw server logs or display each request event. Categories, crawler totals and path details use different groupings and source limits, so they should not be treated as a complete request ledger or expected to reconcile exactly. Missing source data is shown as unavailable.
What counts as a successful or unsuccessful crawler request?
Screpy counts HTTP 2xx responses as successful. Responses outside the 2xx range are counted as unsuccessful for this view, so the drilldown can include redirects as well as client and server errors. Review the exact status before deciding whether a response needs a fix.
Does AI bot traffic prove that my content appeared in an AI answer?
No. The analytics record a verified crawler's request and response during the reporting period. A blocked request can be included, so the record does not even prove the crawler downloaded the content. It does not establish indexing, model training, a brand mention, citation or answer inclusion. Use AI Visibility to examine monitored prompts and stored answers.