Seven-day traffic trend
Review the latest seven days of verified bot requests in hourly buckets instead of judging crawler activity from one isolated visit.
AI Bot Traffic Analytics for Verified Crawler Requests
Connect the Cloudflare zone matched to your Screpy project and review the latest seven days of verified bot activity. Compare crawler categories, publishers, data transfer, request trends, successful responses, and non-2xx URLs without confusing a crawler visit with an AI citation.
TL;DR
Too Long; Didn't Read
Show summary Hide summary
TL;DR
Too Long; Didn't Read
What is Screpy AI & Bot Traffic?
Screpy AI & Bot Traffic is a Cloudflare-connected analytics view for verified crawler requests. It turns the latest seven days of zone activity into category trends, crawler-level totals, transfer volume, response outcomes, and a path-level non-2xx drilldown. It measures access evidence, not AI answer visibility.
Connect the correct zone
- Add an active Cloudflare API token with zone and analytics access.
- Screpy matches available zones to the website domains in your projects.
- The view stays unavailable when the required zone or dataset cannot be read.
- No substitute traffic source is silently used when the connection is incomplete.
Read the seven-day trend
- The current reporting period is fixed to the latest seven days.
- Request volume is grouped into hourly buckets.
- The highest-volume verified bot categories appear as separate series.
- Lower-volume categories are combined into an Other series when needed.
Compare verified crawlers
- Review crawler name, publisher, and verified category.
- Compare bytes transferred and total request volume.
- Separate successful 2xx requests from non-2xx responses.
- Use the category filter and per-crawler sparkline to narrow the review.
Investigate non-2xx paths
- Open the unsuccessful count for the crawler that needs attention.
- Inspect the requested path, exact HTTP status, and request count.
- Treat redirects, client errors, and server errors according to their intent.
- Do not present a crawler request as proof of indexing, training, or citation.
See the crawler evidence the dashboard actually records.
Use traffic volume, identity, category, transfer size, response outcome, and path-level errors together. Each field answers a different part of the crawler-access question.
Verified bot categories
Compare the highest-volume verified bot categories on a stacked chart, with lower-volume categories combined into Other.
Crawler and publisher identity
See a readable crawler name and publisher when Screpy can map the verified request to a known search, AI, monitoring, or social crawler family.
Category filter
Filter the verified crawler table by the categories present in the connected zone and focus the comparison on one class of automated traffic.
Bytes transferred
Measure how much response data each crawler received during the reporting period alongside its request volume and trend.
Successful and non-2xx requests
Separate successful 2xx responses from redirects, client errors, and server errors rather than treating every crawler request as a successful fetch.
Per-crawler request sparkline
Use the hourly request sparkline to see whether one crawler was consistently active or appeared in a short burst during the seven-day window.
Unsuccessful URL drilldown
Open any crawler with non-2xx requests to inspect the affected path, exact HTTP status, and number of unsuccessful requests.
Start with traffic classified as verified bot activity.
Screpy reads the matched Cloudflare zone and limits this view to requests with a verified bot category. That keeps the analysis focused without claiming that every automated visitor has been identified.
- Review seven complete days in hourly buckets
- Compare the top verified bot categories and Other
- Keep crawler evidence tied to the matching website zone
Compare who requested content and how the site responded.
The crawler table combines identity, publisher, category, transfer volume, total requests, 2xx success, non-2xx outcomes, and an hourly sparkline in one review surface.
- Filter the table by available verified categories
- Compare request volume with bytes transferred
- Prioritize crawlers with repeated non-2xx responses
Move from an aggregate failure count to the affected paths.
Select a crawler with unsuccessful requests to see the URL path, HTTP status, and repeated request count. This narrows the technical review without assuming every non-2xx response is an error.
Learn more about non-2xx investigation- Separate redirects, client errors, and server errors
- Find paths repeatedly requested with the same status
- Send unexpected responses into a focused website audit
Turn verified bot traffic into a focused technical review.
The useful outcome is not a larger crawler chart. It is a specific decision about which crawler, category, or non-2xx path deserves investigation.
- 01
Connect Cloudflare and match the project zone
Use an active API token that can list the website zone and read analytics, then confirm the project domain has a matching connected zone.
- 02
Read the seven-day category pattern
Look for sustained activity, sudden bursts, missing expected categories, or a large Other segment before opening individual crawler rows.
- 03
Compare crawler volume and outcomes
Use requests, bytes transferred, hourly trend, successful 2xx responses, and non-2xx counts to choose the crawler that needs attention first.
- 04
Inspect paths and verify the intended response
Review each repeated path and status, confirm whether it is intentional, investigate unexpected responses, and use later traffic evidence to confirm the behavior changed.
Use crawler traffic where it changes a real decision.
The view is most useful when a team owns enough website context to decide whether a traffic pattern or HTTP response is expected.
Publishers and content libraries
See whether verified search and AI crawler families are requesting the site and whether important content paths return non-2xx responses.
SEO and content teams
Add real request evidence to crawlability discussions without presenting crawler activity as proof of indexing, citations, or answer visibility.
Product and engineering teams
Hand off a specific crawler, path, HTTP status, and request count when an application route or deployment needs technical investigation.
Agencies and multi-project teams
Keep each client or website zone connected to its matching Screpy project and use the same review method across separate properties.
Know what crawler traffic cannot prove.
Clear limits make the evidence safer to use in SEO, content, engineering, and AI-search discussions.
A request is not an AI citation
A verified crawler request shows that a bot reached a URL. It does not prove that the page was indexed, used for training, cited, or included in an AI answer.
Seven days, not custom history
The current Screpy view uses a fixed seven-day period with hourly trend data. It does not promise arbitrary date ranges or long-term bot history.
Verified traffic, not every automated request
The view is built from verified bot categories available for the connected zone. It is not a raw server-log explorer or a complete inventory of every scraper.
Analytics, not crawler control
Screpy reports activity and response outcomes. The current view does not block crawlers, change access rules, send bot alerts, or export a full request log.
Choose the AI-search workflow that matches the question.
Use crawler analytics for website requests, AI Visibility for monitored answers, or Website Audit when a repeated response problem needs a broader technical investigation.
Review seven days of verified crawler categories, request trends, transfer volume, response outcomes, and non-2xx paths.
Monitor selected prompts for brand mentions, citations, source domains, answer position, sentiment, competitors, and share of voice.
Crawl the wider site when non-2xx bot requests point to redirects, broken destinations, internal-link problems, or recurring template issues.
Related SEO guides
Learn the concepts behind this workflow and apply them with practical SEO guidance.
GEO
llms.txt: What It Is, How to Create It, and Its SEO Limits
Learn how llms.txt works, how to create a useful file, what it cannot guarantee, and how to measure whether AI crawlers actually use it.
Read article GEO
How Do You Rank on ChatGPT?
Rank on ChatGPT by making pages crawlable, answering user questions clearly, and building credible sources that earn citations in relevant search responses.
Read article SEO
How Does an AI Technical SEO Audit Find Website Issues?
An AI technical SEO audit analyzes crawl and performance data to flag indexation, redirects, broken links, and schema issues, then prioritizes fixes by impact.
Read article SEO
Server-Side Rendering vs Client-Side Rendering for SEO
Server-side rendering vs client-side rendering: compare crawlability, render queues, Core Web Vitals, and hydration to choose SSR, CSR, SSG, or ISR today.
Read articleAI bot traffic analytics questions, answered.
Understand the connected data source, fixed reporting period, verified crawler scope, response logic, URL drilldown, and the difference between crawler access and AI answer visibility.
What does Screpy AI & Bot Traffic measure?
Screpy uses analytics from the Cloudflare zone matched to a project to report the latest seven days of verified bot traffic. The dashboard includes an hourly category trend and a crawler table with publisher, category, bytes transferred, request totals, successful requests, non-2xx requests, and a per-crawler trend.
What is required before bot traffic data appears?
Connect an active Cloudflare API token that can list the relevant zone and read its analytics. Screpy then matches project domains to accessible zones. If the token lacks analytics permission, the project domain does not match a zone, or the required dataset is unavailable for that zone, the dashboard shows an unavailable state instead of substitute data.
Which crawlers can Screpy identify?
The table starts with traffic categorized as verified bots by the connected analytics source. Screpy adds readable identities for known crawler families such as Googlebot, Bingbot, OpenAI, Anthropic, Perplexity, Apple, Meta, and other recognized services when their requests are present. Available rows depend on actual zone traffic.
What counts as a successful or unsuccessful crawler request?
Screpy counts HTTP 2xx responses as successful. Responses outside the 2xx range are counted as unsuccessful for this view, so the drilldown can include redirects as well as client and server errors. Review the exact status before deciding whether a response needs a fix.
Does AI bot traffic prove that my content appeared in an AI answer?
No. A request proves only that a verified crawler reached a URL during the reporting period. It does not prove indexing, model training, a brand mention, a citation, a ranking position, or inclusion in an answer. Use AI Visibility when the question is about monitored prompts and answers.
Is Screpy AI & Bot Traffic a server log analyzer?
No. It reads aggregated analytics for the matched Cloudflare zone and presents a focused verified-crawler view. It does not ingest raw Apache, Nginx, CDN, or application logs, and it does not expose every individual request event.
Can Screpy block AI crawlers or change robots.txt from this page?
No. The current feature is an analytics and investigation view. Access controls, firewall rules, robots directives, crawler alerts, referral analytics, and CSV exports are not managed from this Screpy page.
How should I investigate unsuccessful crawler URLs?
Open the non-2xx count for the affected crawler, review each path and HTTP status, then decide whether the response is intentional. Unexpected redirects, 4xx responses, or 5xx responses can be checked against the live route, internal links, canonical destination, crawl configuration, and a wider Website Audit before a change is made.
See which verified crawlers reach your site—and where requests fail.
Connect the matching Cloudflare zone, review the seven-day crawler pattern, compare response outcomes, and investigate repeated non-2xx paths with the right technical context.