How to Use Crawl Stats to Understand Googlebot Activity

Crawl Stats in Google Search Console is the fastest way to see what Googlebot is requesting from your site and how your infrastructure is responding. It summarizes crawl requests, download size, average response time, and host status, with drill-downs by response codes, file type, and crawl purpose. Use it to spot sudden shifts after a release or migration, confirm whether slowdowns or 5xx errors are forcing Google to throttle, and catch crawl waste on low-value URLs or heavy assets that crowd out important HTML pages. The non-obvious win is reading metrics in pairs, since a higher crawl count can simply mean Googlebot is retrying failed requests.

Where to find Crawl Stats in Google Search Console

Required property type and permissions

Crawl Stats lives inside Google Search Console, but it is not available for every property setup. Google shows the report only for root-level properties, meaning either:

• A Domain property (like example.com or m.example.com), or
• A URL-prefix property at the root (like https://example.com)

If you verified a subfolder property such as https://example.com/blog/, you typically will not see Crawl Stats for that view. In practice, most SEO teams prefer a Domain property because it gives the cleanest, domain-wide picture of Googlebot activity across subdomains and protocols.

On the access side, Crawl Stats is a reporting feature, so you generally just need to be added as a user on the property to view it. Owners have full control (including adding users and changing settings), while Full and Restricted users mainly differ in what actions they can take. If you cannot find the report at all, confirm you are on the right property and that your account has access to it in Search Console settings.

What you see on the first screen

To open the report, go to Settings (Property settings) > Crawl stats. Google also documents the navigation and report structure in its Crawl Stats report help page.

The landing screen is designed for fast triage. You will usually see:

• A 90-day trend view for total crawl requests, total download size, and average response time
• A Host status summary that flags DNS, server connectivity, or robots.txt availability issues
• Breakdown tables for crawl activity by response, file type, crawl purpose, and Googlebot type

If you are using a Domain property with multiple hosts or child domains, you may also see a hosts list, so you can quickly scope the report to a specific host that is showing unusual crawl behavior.

Trend charts in Crawl Stats: requests, download size, response time

Interpreting spikes, drops, and seasonality

Start by reading the three trend lines together, not in isolation. Crawl requests answers “how many URLs Googlebot tried,” while download size answers “how heavy those fetches were.” A requests spike with flat download size often points to lots of lightweight URLs (for example, redirects, parameters, or thin pages). A download size spike with a mild request increase usually means Googlebot is pulling heavier resources, such as large images, JS bundles, CSS, or PDFs.

Common reasons for spikes include a big content release, a sitemap refresh, internal linking changes, a migration, or a redirect rollout that multiplies the number of URLs Googlebot has to touch. Another frequent cause is accidental URL growth from faceted navigation, calendar pages, or tracking parameters.

Drops can be normal, but you should investigate when they are sudden and sustained. Typical causes are server instability, robots.txt mistakes, a new authentication wall, widespread 4xx/5xx errors, or major canonical and noindex changes that reduce the set of URLs Google considers worth crawling.

Seasonality is real. Ecommerce sites often see crawl lift around major shopping periods. News and fast-updating sites can see crawl increase when publication volume rises. Frequently updated sections may be crawled at a different cadence, but Crawl Stats alone cannot prove why Google changed its crawl demand.

Response time changes that signal server strain

Average response time is your early warning for crawl throttling. A one-day bump is often noise. A multi-day step change matters, especially when it aligns with higher 5xx, timeouts, or “other errors” in the response breakdown.

Red flags to watch for:

• Response time rises while requests fall: Googlebot may be backing off because your site is slower or less reliable.
• Response time rises while download size rises: heavy pages or assets may be stressing origin, cache, or database.
• Response time rises after security changes: WAF rules, bot protection, TLS misconfigurations, or aggressive rate limiting can slow or block Googlebot even if users seem fine.

When you spot a shift, compare the date range to deploys, hosting changes, CDN configuration, and application monitoring. Crawl Stats shows what Googlebot experiences, which is often different from what a browser or synthetic test sees.

Host status signals: server connectivity, DNS, and robots.txt

Reading each host status category

The Host status panel is the quickest “can Google even reach me?” check in Crawl Stats. It separates availability into three layers, which helps you diagnose the right problem fast:

• DNS resolution: Googlebot could not reliably resolve your hostname to an IP address. This often points to DNS outages, misconfigured records, or flaky authoritative nameservers. You may also see this when recent DNS changes have not fully propagated, or when one nameserver is failing while others look fine.
• Server connectivity: DNS worked, but Googlebot could not establish a stable connection to your web server. Typical causes include origin downtime, overloaded infrastructure, bad TLS configuration, aggressive rate limiting, or security tooling (WAF, bot protection) that blocks or challenges Googlebot.
• robots.txt fetch: Googlebot could not retrieve /robots.txt. Since robots.txt is the gatekeeper for crawling rules, this one can have an outsized impact even if the rest of the site is healthy.

Host stability matters because repeated connectivity, DNS, or robots.txt failures can prevent reliable crawling. If the host status is red, treat it as a reliability issue first, not a content issue.

Robots.txt fetch failures and their impact

Google generally checks robots.txt before crawling URLs on a host. If robots.txt is temporarily unreachable (for example, it returns 5xx, times out, or is blocked), Google may reduce or pause crawling to avoid accidentally crawling content that might be disallowed. That can look like a sudden crawl drop, slower discovery, and delayed refreshes of updated pages.

Practical fixes that usually move the needle:

• Make sure /robots.txt is publicly accessible and responds quickly, ideally from cache or CDN if appropriate.
• Avoid redirect chains or authentication challenges on /robots.txt.
• Check whether your firewall or bot protection is blocking Googlebot requests to /robots.txt specifically.
• Correlate the dates with incidents, deploys, hosting changes, and WAF rule updates.

If you need a deeper diagnostic checklist for crawl availability errors, Google’s crawling error troubleshooting guide is the most reliable reference.

Crawl response breakdowns: 200, redirects, 4xx, 5xx, and timeouts

Patterns that indicate availability vs URL hygiene issues

The response breakdown is where Crawl Stats turns into a diagnostic tool. It shows how Googlebot’s requests were answered, which helps you separate “the site is down” from “the site is messy.”

Look for these patterns:

• Healthy crawling (mostly 200, low errors): A high share of 200s is normal for an actively crawled site. Some redirects and 4xx will always exist on large sites, but they should be a minority and not rising week over week.
• Availability problems (5xx, timeouts, connection issues): Spikes in 5xx or timeouts usually mean the origin is overloaded, misconfigured, or intermittently failing. If this aligns with rising average response time and falling crawl requests, Googlebot is likely backing off to protect your server.
• URL hygiene problems (4xx, soft 404 behavior, excessive redirects): A rising 4xx share often signals broken internal links, expired inventory pages without proper handling, parameter sprawl, or a sitemap that includes URLs that no longer exist. This is “crawl waste”: Googlebot spends time fetching URLs that cannot be indexed or are low value.

A simple rule: server-side errors are urgent reliability issues, while 4xx and redirect bloat are usually architecture and process issues. Both matter, but the remediation path is different.

In AI-era SEO, URL hygiene is also about clarity. When Googlebot repeatedly hits low-value variants (parameters, sort orders, internal search URLs), you make it harder for Google to focus crawl and indexing attention on the canonical pages you want surfaced in AI summaries and rich results.

Redirect-heavy crawling and chain problems

Redirects are not inherently bad. They are essential for migrations, canonical host enforcement, and consolidating duplicates. The problem is when redirects become the dominant response type, or when Googlebot keeps encountering the same redirects because your internal links, sitemaps, and hreflang still point to old URLs.

What to watch for:

• High redirect volume after a launch: normal for a short period, but it should trend down as you update internal links and sitemaps.
• Redirect chains (A → B → C): they add latency, increase crawl cost, and can amplify response time issues. Chains also increase the odds of losing signals if something breaks mid-chain.
• Redirect loops (A → B → A): these waste crawl and can prevent indexing entirely.

Best practice is simple: aim for one hop wherever possible. Update internal links to the final destination, refresh XML sitemaps to only include canonical 200 URLs, and remove legacy redirect layers once traffic and crawling stabilize after a migration.

Crawl segmentation: file type, crawl purpose, and Googlebot type

Discovery vs refresh signals and what they imply

Crawl Stats lets you segment by crawl purpose, which is one of the most actionable views for technical SEO. In plain terms:

• Discovery crawling is Googlebot looking for new URLs. Increases here often follow new internal links, updated sitemaps, improved navigation, or a surge in new content.
• Refresh crawling is Googlebot revisiting URLs it already knows to check for changes. Spikes here can be a good sign for frequently updated content, but they can also indicate instability (Googlebot rechecking because responses are inconsistent).

If discovery is low on a site that is publishing, the issue is often structural: weak internal linking, orphan pages, inconsistent canonicalization, or sitemaps full of non-canonical URLs. If refresh is high but index coverage is not improving, look for slow responses, redirect churn, or pages that change URLs and templates too often.

For AI-era SEO, refresh behavior is especially important on pages where accuracy changes quickly (pricing, availability, policies, specs). If Google cannot refresh reliably, it is less likely to treat those pages as dependable sources for prominent AI-driven surfaces.

Googlebot smartphone vs desktop differences

Google reports crawling by Googlebot type, commonly split into smartphone and desktop. For most sites, smartphone crawling dominates because Google primarily uses the mobile version for crawling and indexing. When you see a mismatch, treat it as a diagnostic clue, not a score.

Useful interpretations:

• Smartphone crawl drops but desktop stays steady: mobile delivery might be slower, blocked, or returning different status codes.
• Smartphone response time is worse: mobile templates, client-side rendering, or edge rules may be heavier than expected.

Google’s overview of Googlebot helps clarify the different crawler types and how they behave.

When resource file crawling is the real bottleneck

The file type breakdown is where many “mysterious crawl budget” problems become obvious. If a large share of crawling is images, JavaScript, CSS, or media files, Googlebot may be spending capacity on assets instead of HTML pages that actually drive indexing and rankings.

This often happens after a redesign that ships larger bundles, unoptimized images, or multiple third-party tags. The fix is usually performance and hygiene work: compress and resize images, reduce unused JS/CSS, cache static assets aggressively, and eliminate duplicate asset URLs. When resource crawling becomes leaner, HTML crawling and recrawl frequency often stabilize too.

Normal vs suspicious Crawl Stats patterns to watch

Sudden crawl drop with stable traffic

A crawl drop is not automatically a problem. Googlebot adjusts crawl rate based on perceived need and on your site’s ability to respond. If your content is relatively stable and your internal linking does not change much, it is normal to see crawling soften over time.

It becomes suspicious when the crawl drop is sharp and sustained, but search traffic and rankings look stable. That pattern often means Google can still serve your already-indexed pages, but it is struggling to refresh and discover updates. Common causes include:

• A new CDN, WAF, or bot mitigation rule that slows or intermittently blocks Googlebot.
• Increased response times due to database load, cache misses, or heavier pages.
• robots.txt fetch issues that make Google cautious.
• A shift to JavaScript-heavy rendering that makes pages slower to fetch and process.

The practical test is to line up dates. If the crawl drop starts the same day as an infrastructure or security change, treat it as an engineering incident, even if users do not complain.

Error mix shifts after releases or migrations

The most valuable Crawl Stats habit is watching the error mix, not just total errors. After a release, redesign, or migration, small issues can cascade into large crawl waste.

Patterns worth acting on quickly:

• Redirect share rises and stays high: internal links and sitemaps are still pointing at old URLs, or you introduced unnecessary redirect layers (HTTP to HTTPS, non-www to www, then locale routing, etc.).
• 4xx rises after a content change: navigation or templating may be generating broken links, parameterized URLs, or out-of-stock URLs that now return 404 without a replacement.
• 5xx or timeouts rise after a deployment: capacity, caching, or application errors are impacting Googlebot. Even short-lived spikes can slow recrawling and delay index updates.
• Errors concentrate on one host or subdomain: a partial migration or isolated backend is failing, while the rest of the site is fine.

For modern SEO teams, this is also an AI-readiness check. AI-driven search features reward consistency. If Googlebot keeps hitting redirect churn, intermittent 5xx, or unstable templates, your site can look less reliable as a source, even when the content is strong.

Limits of Crawl Stats and when to use other data

What the 90-day window and aggregation hide

Crawl Stats is a trend and diagnostics report, not a forensic one. You typically get a rolling 90-day view, which is great for spotting changes after releases, but it can hide slower seasonal patterns or long-running crawl inefficiencies.

More importantly, Crawl Stats is aggregated. It tells you counts and percentages, but it does not tell you the exact URLs that produced each status code, which templates were slowest, or which parameter combinations created crawl waste. That means you can identify “we have a redirect problem” or “5xx spiked,” but you often cannot answer “which 1,000 URLs caused it?” from this report alone.

It also only reflects Google’s crawlers. Crawl Stats does not report requests from unrelated AI crawlers or tool-specific fetchers. For that broader picture, you need server-side data.

When server logs are necessary

Use server logs when you need precision, such as:

• Pinpointing which URLs return 404/500 to Googlebot
• Finding redirect chains and repeated crawl traps (filters, calendar URLs, internal search)
• Measuring true response times and bytes by endpoint, not averages
• Verifying whether Googlebot is being blocked or challenged by a WAF/CDN rule
• Auditing non-Google crawlers (including AI and LLM-related user agents) that Crawl Stats cannot report on

Logs turn “something changed” into “this specific route and rule broke.”

Connecting Crawl Stats to Page indexing and URL Inspection

Think of Crawl Stats as the “weather report,” then use other Search Console views for the “street-level view.” If crawl requests are dropping or errors are rising, check the Page indexing report to see whether indexing outcomes changed at the same time, and what reasons Google is giving site-wide. The official Page indexing report documentation is worth keeping bookmarked.

Then use URL Inspection on a few representative URLs (a top template, a recently updated page, and a problematic example) to confirm the canonical URL Google chose, whether the page was crawled recently, and whether Google could fetch it successfully.

Turn Crawl Stats into an investigation queue

Crawl Stats is most useful when each anomaly leads to a specific verification step.

• Requests fall after a deployment: verify host availability and robots.txt, then crawl with the Screpy SEO Crawler to find navigation, status-code, canonical, or rendering changes.
• Average response time rises: group the change by host and file type, compare it with server monitoring, and inspect whether slow HTML responses coincide with more 5xx or timeout responses.
• Redirect or error responses grow: identify representative URLs, trace the final target, and remove redirected or broken URLs from internal links and sitemaps.
• Discovery requests rise: review newly introduced URLs, faceted navigation, parameters, and calendar-like paths before treating the increase as healthy growth.
• Crawl looks stable but search performance changes: use the Screpy Search Console dashboard to compare clicks, impressions, pages, and queries. Crawl activity and search visibility answer different questions.

Record the date range, property, host, affected response category, sample URLs, suspected release, and verification result. This prevents a normal short-term fluctuation from becoming an unsupported crawl-budget conclusion.