Crawl Failure Hotspots

Learn how to investigate repeatedly requested URLs that return failed responses and prioritize fixes by status, frequency, source, and SEO impact.

Crawl Failure Hotspots identifies URL paths repeatedly requested by crawlers that return unsuccessful responses. Screpy combines the path, response status, crawler-request volume, and failed-request count so frequent operational problems rise above isolated errors.

A failure is important when it prevents useful content from being crawled, wastes repeated requests, breaks internal journeys, or signals a shared infrastructure problem. Not every crawler request should return a page; deliberately unavailable or abusive URLs can be valid failures.

Read the report

Field	Meaning
URL / path	The requested resource or pattern
Status	The HTTP response category returned to the crawler
Crawler requests	How often identified crawlers requested the path
Failed requests	Requests that received an unsuccessful response

Prioritize repeated failures on important URLs, template-wide patterns, and server errors before low-volume requests for nonexistent junk paths.

Understand status categories

4xx: the requested resource was not served

404 / 410: The URL is missing or intentionally gone. This may be correct for removed content, but internal links and sitemaps should not keep generating requests.
401 / 403: Access is restricted. Confirm whether important public pages are blocked by authentication, firewall, bot rules, or security settings.
429: The requester is being rate limited. Repeated 429 responses can prevent a crawl from completing; review limits and allow the Screpy crawler where appropriate.

5xx: the server could not complete the request

Server errors can be transient, load-related, application-specific, or caused by upstream dependencies. Repeated 5xx responses on canonical pages require urgent investigation because content availability is unreliable.

Investigate step by step

Group rows by status code and URL pattern.
Identify whether the URL should exist and be publicly crawlable.
Test the final canonical URL without relying only on a browser's cached result.
Inspect internal links, sitemaps, canonicals, hreflang, scripts, and templates that generate the failed path.
Compare failure timing with deployments, traffic spikes, security-rule changes, and rate limits.
Decide whether to restore, redirect, remove references, allow access, or intentionally keep the failure.
Verify both the response and the source that generated the request.

Choose the right response

Situation	Preferred direction
Valuable page accidentally returns 404	Restore it or redirect to the closest true replacement
Removed page has no equivalent	Keep a clear 404/410 and remove internal references
Internal URL redirects then fails	Update the source link and repair the final destination
Public page returns 403 to legitimate crawlers	Review firewall and bot access rules
Important pages intermittently return 5xx	Diagnose capacity, application, and dependency failures
Random exploit path returns 404	Usually no content fix; monitor security patterns separately

Example

The report shows thousands of requests to /category/widgets?page=2 returning 500. Similar pagination URLs also fail. This is not a one-page issue; the URL pattern or template is broken. Fix the pagination behavior, verify representative pages, and ensure internal links do not generate invalid values.

By contrast, repeated requests to /wp-login.php on a non-WordPress site do not justify creating or redirecting that page.

False positives and cautions

Security scanners and malicious bots may request irrelevant paths.
A removed URL can correctly return 404 or 410.
A temporary failure during deployment may no longer reproduce.
One path may represent many query-string variations.
Returning 200 with an error message creates a soft-error problem that status-only review can miss.

Verify the resolution

Run a new Screpy crawl for site-level access issues and review fresh request data over a comparable period. Confirm failed-request volume falls and internal sources no longer generate invalid URLs. Do not call the issue resolved only because one manual request succeeds.