Crawl Cost Analysis

Most Expensive URLs ranks paths responsible for high data transfer. Screpy shows request count, total transferred bytes, and bytes per request so you can separate popular resources from unusually heavy responses.

This report is not a hosting bill. It is a prioritization view for understanding where crawler and visitor requests consume transfer and processing resources.

Read the three core metrics

The relationship matters:

• High requests + moderate weight can create high total transfer.
• Low requests + very high weight identifies oversized individual responses.
• High requests + high weight is usually the strongest efficiency candidate.

Classify expensive URLs

Popular, necessary pages

A high-traffic landing page can be expensive because it is successful. Optimize carefully and protect functionality, content, and conversion behavior.

Oversized HTML or API responses

Large markup, embedded state, duplicate data, or unbounded responses increase transfer and parsing work. Measure the actual payload before removing useful content.

Heavy media and downloads

Images, documents, video, fonts, and archives may be legitimate but poorly compressed, incorrectly sized, or repeatedly transferred without effective caching.

Cache misses

Frequently requested static or stable resources should not require full origin generation every time. Review cache headers, variation keys, cookies, and query parameters.

Trap or attack traffic

Unbounded parameters, scraper behavior, and invalid routes can consume transfer without user value. Join the result with Crawl Trap Candidates and Crawl Failure Hotspots.

Analyze step by step

1. Sort by total transfer to find the largest aggregate impact.
2. Sort mentally by bytes per request to find oversized responses.
3. Identify the content type and whether the URL is user-facing, crawler-facing, or an asset.
4. Check request frequency, cache status, compression, response headers, and parameter variants.
5. Group paths by template or resource type.
6. Estimate business and crawl importance before optimizing.
7. Fix the shared cause rather than compressing isolated rows one by one.

Optimization directions

• Compress text responses with an appropriate content encoding.
• Resize and encode images for their rendered use.
• Remove duplicate or unnecessary response data.
• Paginate or bound large dynamic responses.
• Cache stable resources with correct invalidation.
• Normalize parameter variants that defeat caches.
• Stop generating invalid or trap URLs.
• Serve downloads intentionally instead of embedding large payloads in HTML.

Do not remove useful page content merely to reduce transfer. Performance optimization should preserve the user's task.

Example

An image path accounts for the most transfer. Request count is moderate, but bytes per request is 8 MB. The image is displayed at 800 pixels wide yet delivered as a large uncompressed source file.

The direct action is responsive sizing and modern compression. If the same total transfer came from a 40 KB file requested millions of times with cache misses, caching and request behavior would be the first investigation instead.

Connect crawl cost to SEO

Efficient responses improve reliability and can help large sites make better use of crawling resources. For small and medium sites, do not overstate crawl budget: focus first on user experience, server stability, duplicate URL control, and avoiding infinite spaces. Google's crawl-budget documentation is primarily relevant to large or rapidly changing sites.

Verify the result

Measure response size, compression, cache behavior, and functionality after release. Compare total transfer and bytes per request over equivalent traffic periods. If request volume changed, normalize the comparison; a lower total caused only by fewer visitors is not an efficiency improvement.