Allow the Screpy crawler through a firewall or WAF
Allow the Screpy crawler IP, configure firewall and bot-protection rules, and fix blocked or incomplete website analyses.
Firewalls, WAFs, CDNs, hosting security tools, and WordPress plugins can block website crawlers even when the site works normally in a browser. Allow both Screpy's crawler IP and its user-agent before starting an analysis.
Screpy crawler access details
Allow IP address 5.75.247.29. The crawler identifies itself with the user-agent ScrepyBot. Apply the rule to the exact hostname and every security layer protecting it.
How to allow the Screpy crawler IP
Find every active security layer
Check your CDN or WAF, hosting control panel, server firewall, reverse proxy, application middleware, and security plugins. Write down which layer can block, challenge, or rate-limit requests.
Add the IP to an allow rule
Create an allow or skip rule for 5.75.247.29. Where supported, skip managed firewall checks, bot challenges, CAPTCHA, browser integrity checks, and request-rate rules for this IP.
Allow the crawler user-agent
Ensure custom bot rules do not reject ScrepyBot. An IP allow rule and a user-agent rule solve different filters, so check both.
Apply the rule to the correct host
Confirm that the rule covers the exact project hostname, including www or a subdomain. If traffic passes through more than one proxy or firewall, repeat the allow rule where necessary.
Verify and run a new analysis
Review security events or server access logs for the crawler request. After the change is active, start a new analysis; a completed crawl will not update retroactively.
What a successful allow rule must do
The crawler should receive the real page HTML with a successful response. These outcomes still indicate a block:
- HTTP
401,403,429,500,502,503, or504. - A CAPTCHA, browser check, or “verify you are human” page.
- A
200 OKresponse containing only a challenge script or access-denied message. - A connection timeout before the website sends content.
- Different content based on country, ASN, data-center network, referrer, or user-agent.
Use logs to identify the blocking layer
Search firewall and server logs for requests from 5.75.247.29. No request in the origin log often means an upstream CDN, WAF, or network rule stopped it first. A request with 403 or 429 usually points to the layer that returned that response.
If the IP is allowed but Screpy is still blocked
Check whether bot management, CAPTCHA, rate limiting, country rules, ASN rules, or a hosting security product runs before or after the firewall rule. Some systems require an explicit “skip security checks” action rather than a simple allow action.
Confirm the project hostname and DNS destination first. Then check the CDN or reverse proxy logs. If the request never reaches the origin server, the block is likely upstream of the host.
Inspect the response body served to automated traffic. A challenge page, empty HTML shell, content-protection script, or referrer-dependent response can return 200 without containing the real page.
Add an exception for 5.75.247.29. Country access for normal visitors does not automatically permit data-center traffic or the crawler's network.
Exclude the crawler IP from request-rate rules when possible. Otherwise reduce crawler concurrency and add a delay in Project settings, then start a new analysis.
Next, confirm that robots.txt allows ScrepyBot. If the analysis still fails or finds only a few pages, follow the crawl troubleshooting decision guide.
Website analysis checklist before adding a Screpy project
Check website access, HTTPS, redirects, firewall rules, robots.txt, bot protection, and rate limits before starting a Screpy crawl.
Configure robots.txt for the Screpy crawler
Allow ScrepyBot in robots.txt, fix blocked website analyses, and understand conflicting User-agent, Allow, and Disallow rules.