HTTP Status Codes and Crawl Errors

HTTP status codes tell browsers and crawlers what happened when a URL was requested. In Pages, use Success (2xx) to confirm accessible URLs, Client Errors (4xx) to find requests the server rejected or could not satisfy, and Server Errors (5xx) to find requests the server failed to process.

Read the response before choosing a fix

• 200 OK: the server returned a successful response. It does not prove that the content is correct, indexable, or canonical.
• 204 No Content: successful but intentionally empty; unusual for an indexable HTML page.
• 401/403: authentication or access rules prevented the request. This may be intended for private areas but serious for public pages.
• 404 Not Found: the URL is unavailable. It can be correct for deleted content, but internal links and sitemap entries should not keep requesting it.
• 410 Gone: explicitly indicates deliberate removal.
• 429 Too Many Requests: rate limiting rejected the crawl. Review crawler access and server limits before concluding the page is broken.
• 500/502/503/504: the application or upstream server failed, was unavailable, or timed out.

Google treats 4xx URLs other than 429 as unavailable and eventually removes previously indexed versions; 5xx and 429 responses cause crawling to slow. Use Google's HTTP status documentation when evaluating search behavior.

Analyze the pattern

1. Confirm the status on the public URL, not only in a CMS preview.
2. Check whether the URL should exist and be discoverable.
3. Find internal links, sitemap entries, canonical tags, or hreflang references that still use it.
4. Group failures by path, template, timestamp, and status.
5. Compare one failed request with a successful URL from the same template.

A single 404 after intentional deletion may need no redirect. Hundreds of product URLs returning 500 usually indicate a shared application or data problem. A site-wide 403 or 429 pattern can indicate firewall or rate-limit rules blocking Screpy rather than page removal.

HTTP/1.1 404 Not Found
Content-Type: text/html; charset=utf-8

Choose the correct action

• Restore the page when the URL should still serve content.
• Redirect only when a clear replacement satisfies the same purpose.
• Keep a genuine 404 or 410 when content was removed without a replacement.
• Remove failed URLs from internal links and sitemaps.
• Investigate logs and shared dependencies for repeated 5xx responses.
• Allow Screpy's crawler through security controls when public pages are incorrectly blocked.

Do not redirect every missing URL to the homepage. That hides broken architecture and can be interpreted as a soft 404.

Example decision

An old campaign URL returns 404, is absent from navigation, and has no equivalent current page. Keep the 404 and remove stale sitemap references. If an internal navigation link still points to it, update the link. If the campaign was permanently replaced by a closely equivalent page with useful external links, use a permanent redirect instead.

Verify the fix

Request representative URLs again, run a new crawl, and confirm the expected status appears in the new Pages snapshot. Check that sitemaps, internal links, canonical targets, and redirect destinations no longer lead to failures.

Diagnose status codes by URL purpose

The same response can be correct or harmful depending on the URL. A deleted campaign page may intentionally return 404. A checkout page returning 404 is urgent. A 200 response is not automatically healthy if the page contains an error message, empty template, or soft-404 experience.

Group URLs by page type, path pattern, and internal importance. Then verify:

• whether the status is stable across repeated requests;
• whether browsers and crawlers receive the same response;
• whether the response body matches the status;
• whether internal links and sitemaps still reference the URL;
• whether a 5xx pattern aligns with load, deployment, or application logs.

How the filters should change

After repairing an accidental 404, Client Errors should fall and Success may rise. If you permanently redirect the URL, Client Errors should fall while Redirects can increase. Removing every internal reference may reduce the crawled URL count instead.

For 5xx errors, a successful recrawl should show stable 2xx responses. One clean request is weak evidence if the failure is intermittent. Sample different times and URL types, and monitor server-side error rates.

Decision example

A discontinued product with no replacement can return 404 or 410 after useful navigation is provided. A product that moved should redirect to the closest equivalent. A misspelled internal URL should be corrected at its source, even when a redirect protects old external links.

MDN explains the meaning of HTTP status-code classes. Treat Screpy's status filters as a triage layer, then judge each response against the URL's intended lifecycle.