Crawl Cost Analysis

Most Expensive URLs ranks paths responsible for high data transfer. Screpy shows request count, total transferred bytes, and bytes per request so you can separate popular resources from unusually heavy responses.

This report is not a hosting bill. It is a prioritization view for understanding where crawler and visitor requests consume transfer and processing resources.

Read the three core metrics

The relationship matters:

• High requests + moderate weight can create high total transfer.
• Low requests + very high weight identifies oversized individual responses.
• High requests + high weight is usually the strongest efficiency candidate.

Classify expensive URLs

Popular, necessary pages

A high-traffic landing page can be expensive because it is successful. Optimize carefully and protect functionality, content, and conversion behavior.

Oversized HTML or API responses

Large markup, embedded state, duplicate data, or unbounded responses increase transfer and parsing work. Measure the actual payload before removing useful content.

Heavy media and downloads

Images, documents, video, fonts, and archives may be legitimate but poorly compressed, incorrectly sized, or repeatedly transferred without effective caching.

Cache misses

Frequently requested static or stable resources should not require full origin generation every time. Review cache headers, variation keys, cookies, and query parameters.

Trap or attack traffic

Unbounded parameters, scraper behavior, and invalid routes can consume transfer without user value. Join the result with Crawl Trap Candidates and Crawl Failure Hotspots.

Analyze step by step

1. Sort by total transfer to find the largest aggregate impact.
2. Sort mentally by bytes per request to find oversized responses.
3. Identify the content type and whether the URL is user-facing, crawler-facing, or an asset.
4. Check request frequency, cache status, compression, response headers, and parameter variants.
5. Group paths by template or resource type.
6. Estimate business and crawl importance before optimizing.
7. Fix the shared cause rather than compressing isolated rows one by one.

Optimization directions

• Compress text responses with an appropriate content encoding.
• Resize and encode images for their rendered use.
• Remove duplicate or unnecessary response data.
• Paginate or bound large dynamic responses.
• Cache stable resources with correct invalidation.
• Normalize parameter variants that defeat caches.
• Stop generating invalid or trap URLs.
• Serve downloads intentionally instead of embedding large payloads in HTML.

Do not remove useful page content merely to reduce transfer. Performance optimization should preserve the user's task.

Example

An image path accounts for the most transfer. Request count is moderate, but bytes per request is 8 MB. The image is displayed at 800 pixels wide yet delivered as a large uncompressed source file.

The direct action is responsive sizing and modern compression. If the same total transfer came from a 40 KB file requested millions of times with cache misses, caching and request behavior would be the first investigation instead.

Connect crawl cost to SEO

Efficient responses improve reliability and can help large sites make better use of crawling resources. For small and medium sites, do not overstate crawl budget: focus first on user experience, server stability, duplicate URL control, and avoiding infinite spaces. Google's crawl-budget documentation is primarily relevant to large or rapidly changing sites.

Verify the result

Measure response size, compression, cache behavior, and functionality after release. Compare total transfer and bytes per request over equivalent traffic periods. If request volume changed, normalize the comparison; a lower total caused only by fewer visitors is not an efficiency improvement.

Define useful crawl work

Crawl cost is not simply the number of requests. A request is useful when it discovers, verifies, or refreshes content that matters. Repeated redirects, parameter variants, duplicate pages, large error clusters, and unnecessary resources consume capacity without equivalent value.

Segment requests by status, content type, path family, depth, and change frequency. Compare important templates with low-value patterns so optimization does not accidentally hide useful content.

What improvement looks like

After controlling duplicate URL generation and updating internal links, the share of successful requests to preferred pages should increase. Total requests may fall, error and redirect shares should decline, and important pages may be reached earlier.

If crawl volume drops but newly published pages are discovered more slowly, the controls are too broad or internal linking weakened. If cost rises after a launch, check whether the increase matches legitimate new inventory.

Google notes that crawl-budget management is mainly relevant to very large or rapidly changing sites in its crawl budget guide. Smaller sites should still remove broken architecture, but they do not need to manufacture a crawl-budget problem.

Related guidance

Return to the Quick Wins overview to compare crawl cost with other technical opportunities. Google's robots.txt introduction explains how crawler access rules interact with discovery and crawling.