Fix Screpy crawl and website analysis problems

Use this guide when a Screpy analysis fails, remains incomplete, reports a firewall or robots.txt problem, or crawls fewer pages than expected. Start with the symptom you can observe and change one cause at a time.

Quick crawl diagnosis

The analysis does not start

Check these causes in order:

1. Another crawl is already queued or running.
2. A successful manual analysis started within the last 10 minutes.
3. Website Audit is unavailable for the current plan or disabled in project settings.
4. Project setup did not finish or the project limit was reached.
5. The website cannot be reached from the public internet.

Wait for a running analysis to finish. If Screpy shows a remaining retry time, wait for that window rather than submitting repeatedly. Then confirm Project settings and start one new analysis.

Screpy reports a firewall or access error

An access warning means the crawler did not receive a usable website response. It does not prove that one specific product is responsible.

• Allow 5.75.247.29 in the CDN/WAF, host firewall, server firewall, and security plugins.
• Skip CAPTCHA, managed bot checks, browser challenges, and rate limits for that IP.
• Check country, ASN, data-center, user-agent, and referrer rules.
• Search security events first, then origin access logs.
• Confirm the response body contains the real page, not only a successful status code.

Use the complete firewall and crawler allowlist instructions.

robots.txt blocked the analysis

Open the public /robots.txt file and look for rules that block ScrepyBot, /, or the affected section. Add the following group, publish it, clear relevant caches, and verify the public file:

User-agent: ScrepyBot
Allow: /

Then run a new analysis. See the robots.txt troubleshooting guide for overlapping rules and platform caches.

The crawl finds only one page or too few pages

When the homepage works but expected pages are missing, use this order:

1. Check the completed total: make sure the crawl is not still running.
2. Check access across the site: category or application paths may have different firewall or robots rules.
3. Check internal links: important pages must be reachable through normal crawlable links.
4. Check maximum URLs: the crawl stops adding pages when it reaches the selected capacity.
5. Check depth: deeply nested pages may sit beyond the configured link depth.
6. Check redirects and canonical hosts: links may send the crawler outside the project host or into a loop.
7. Check JavaScript navigation: enable JavaScript rendering only if links or content do not exist in the initial HTML.
8. Check page responses: a security tool may return empty or challenged HTML after the homepage.

The crawl stops partway through

A crawl can stop before the expected total because it reached its URL capacity, exhausted discoverable links, encountered repeated blocks or timeouts, or was rate-limited.

• Compare the completed page count with Max URLs.
• Review whether the missing section is linked from crawled pages.
• Look for bursts of 403, 429, or 5xx responses in server logs.
• Reduce concurrency and add a delay if the server limits request volume.
• Resolve slow origin responses before increasing the scope.
• Start a new analysis after changing access or crawler settings.

The website rate-limits Screpy

Rate limiting usually appears as HTTP 429, temporary access blocks, or a crawl that slows and stops after a burst of requests.

The preferred fix is to exclude 5.75.247.29 from rate-limit rules. If that is not possible:

1. Open the project's Crawler settings.
2. Reduce Concurrency so fewer requests run at once.
3. Add a Delay between requests.
4. Save the settings.
5. Run one new analysis and monitor server logs.

Lower settings make a crawl take longer but reduce pressure on the website. See crawler settings explained.

HTTP errors and timeouts

Results look wrong even though the crawl completed

Before treating a finding as incorrect:

• Select the latest completed crawl.
• Open the exact URL reported by Screpy.
• Check whether the issue existed in the HTML received during that crawl.
• Remember that a browser may show personalized, cached, or JavaScript-rendered content.
• Run a new crawl after changing the website; historical snapshots do not update.
• For broken external links, remember that the destination can block automated requests or rate-limit checks while still opening in your browser.

Questions about failed and incomplete crawls

Once the crawl completes with the expected coverage, continue to Crawling your website to interpret the crawl and Quick Wins to prioritize fixes.