Website analysis checklist before adding a Screpy project
Check website access, HTTPS, redirects, firewall rules, robots.txt, bot protection, and rate limits before starting a Screpy crawl.
Use this checklist before adding a website to Screpy or when an analysis will not start. It separates access problems from crawl-scope problems and prevents repeated failed analyses.
Complete the access checks first
Increasing maximum URLs or crawl depth cannot fix a website that blocks the crawler. First allow 5.75.247.29, check robots.txt, and remove challenges that replace the real page.
Pre-crawl website checklist
- The preferred HTTPS homepage opens without a login.
- The homepage does not show a maintenance page, CAPTCHA, cookie wall, or browser challenge.
- The entered domain reaches the final website without a redirect loop.
-
5.75.247.29is allowed in the firewall, WAF, CDN, hosting panel, and security plugins. - Country, ASN, data-center, and bot rules allow the crawler.
-
https://yourdomain.com/robots.txtis public and does not blockScrepyBot. - Important pages return their real HTML, not an empty shell or challenge script.
- Internal navigation uses crawlable links to important sections.
- Server rate limits can accept the selected concurrency and delay.
- The project URL is the public canonical host you want to analyze.
Verify the website in the right order
Open the exact project URL
Test the URL you plan to enter, including its protocol and hostname. If it redirects, use the final public destination as the project URL.
Check security layers
A website can have several independent filters. Check the CDN or WAF, hosting firewall, server firewall, WordPress security plugin, and custom application rules. Allowing the IP in one layer does not bypass another layer.
Read the public robots.txt file
Open /robots.txt directly. Confirm that a broader rule does not block / or an important directory. Use the Screpy robots.txt examples when rules overlap.
Check rate limiting
Look for HTTP 429, temporary blocks, or bot-protection events in your security and server logs. If the site has strict request limits, reduce concurrency and add a delay in Project settings.
Create the project
When every check passes, continue to Add a project. The first crawl will test the same access path across the site.
Common questions before a website crawl
No. Your browser may already have cookies, pass a CAPTCHA, originate from an allowed country, or be treated as a normal visitor. Screpy needs direct access from its crawler IP and must receive the real page content without a challenge.
Not always. Some security tools return 200 OK with a CAPTCHA, JavaScript challenge, empty template, or access-denied message. The response body must contain the same meaningful page content that a normal visitor receives.
Enable it only when important content or navigation does not exist in the initial HTML. JavaScript rendering takes longer and does not solve firewall, robots.txt, or rate-limit blocks.
A sitemap can help describe important URLs, but it does not replace crawlable internal links or access permissions. Screpy still needs to open every discovered page successfully.
If any check fails, use the crawler troubleshooting guide instead of repeatedly starting new analyses.