Configure robots.txt for the Screpy crawler
Allow ScrepyBot in robots.txt, fix blocked website analyses, and understand conflicting User-agent, Allow, and Disallow rules.
The robots.txt file tells crawlers which parts of a website they may request. A single broad Disallow rule can prevent Screpy from reading the homepage or discovering the rest of the site.
Check the public file, not only your CMS setting
Open https://yourdomain.com/robots.txt in a private browser window. CDN caching, plugins, deployment settings, and hosting tools can serve a different file from the one shown in your editor.
Recommended robots.txt rule for Screpy
Add a dedicated group for Screpy:
User-agent: ScrepyBot
Allow: /This rule allows ScrepyBot to crawl the website. Keep any rules required for other crawlers in their own groups.
How to check robots.txt before an analysis
Open the robots.txt URL
Visit https://yourdomain.com/robots.txt. It should return a readable text file without a login, redirect loop, firewall page, or server error.
Find rules that apply to ScrepyBot
Look for a User-agent: ScrepyBot group. Also inspect broad User-agent: * rules and confirm that important paths are not accidentally blocked.
Add or correct the ScrepyBot group
Add Allow: / under User-agent: ScrepyBot. Remove a Screpy-specific Disallow: / or narrower rules that block sections you want analyzed.
Publish and clear relevant caches
Save the change, publish it, and clear the CDN or page cache if it stores robots.txt. Reopen the public URL to verify the live response.
Start a new crawl
Robots changes affect future requests. Start a new analysis after the public file is correct.
Common robots.txt mistakes
| Rule or symptom | What it means | What to do |
|---|---|---|
User-agent: ScrepyBot followed by Disallow: / | Screpy is blocked from the whole site. | Replace it with Allow: / if the full site should be analyzed. |
| An important directory is disallowed | Pages under that path may be missing. | Remove or narrow the blocking rule. |
/robots.txt returns 403 or a challenge | Screpy cannot reliably read the policy. | Fix firewall access and allow 5.75.247.29. |
| The editor and public file differ | A cache, plugin, or deployment is serving an older file. | Purge the relevant cache and verify the public URL again. |
| Rules changed but the old crawl still looks blocked | Completed crawl data is a snapshot. | Run a new analysis after publishing the fix. |
robots.txt questions
Yes. Use a dedicated User-agent: ScrepyBot group with Allow: /, then keep the restrictions for other user-agent groups. Verify the final public file after editing.
No. Robots rules are crawl instructions, not access control. Protect private pages with authentication and do not add them to a public Screpy project.
robots.txt is only one access layer. A firewall, WAF, bot challenge, rate limit, login, timeout, or invalid response can still block the analysis. Continue with the crawler IP and firewall guide.
Yes. Screpy may analyze the accessible pages while omitting URLs under a disallowed path. Compare the missing section with the rules and internal links before increasing crawl depth.
If the public rules are correct but pages are still missing, use Why Screpy crawls only one page or misses pages.
Allow the Screpy crawler through a firewall or WAF
Allow the Screpy crawler IP, configure firewall and bot-protection rules, and fix blocked or incomplete website analyses.
How to add a project in Screpy
Add a website to Screpy, choose the correct domain and crawl settings, allow crawler access, and fix project setup errors.