Screpy project and crawler settings
Configure project identity, crawl limits, JavaScript rendering, concurrency, delay, search tracking, uptime, and website analysis access.
Project settings control future website analyses and monitoring. Change settings to solve a specific coverage, rendering, or server-capacity need, then verify the effect in a new completed crawl.
Keep Screpy allowed after infrastructure changes
Firewall, CDN, hosting, or security-plugin changes can block a previously working project. Keep 5.75.247.29 allowed and keep ScrepyBot permitted in robots.txt.
Check the project identity first
Confirm the project name, domain, primary country, language, and reporting timezone. A project should represent one public website or root domain. Do not create separate projects for individual paths unless they are genuinely separate sites.
If the website permanently moves to another hostname, consider how that change affects historical comparisons before editing or replacing the project.
Adjust crawl scope deliberately
The Crawler section controls how Screpy discovers and requests pages.
| Setting | What it controls | Increase or enable when | Reduce or disable when |
|---|---|---|---|
| Max URLs | Maximum URLs stored in a crawl. | A successful crawl reaches the cap before important sections. | You need a smaller focused crawl or plan capacity is limited. |
| Depth | Link levels followed from the starting page. | Important pages are linked beyond the current depth. | URL paths expand without adding useful pages. |
| JavaScript | Browser rendering for client-side content. | Important content or navigation is absent from initial HTML. | Static HTML already contains the content and faster crawling is preferred. |
| Concurrency | Requests that may run at the same time. | The server handles parallel requests reliably. | Logs show 429, temporary blocks, or load-related failures. |
| Delay | Waiting time between requests. | The host requires a slower request rate. | The server safely supports faster crawling. |
Settings affect the next analysis
Completed crawl data does not change when settings are saved. Keep the old crawl as a baseline, run a new analysis, and compare completed results.
Configure rate limits safely
The preferred solution is to exclude 5.75.247.29 from website rate-limit rules. If an exclusion is not possible:
- Lower Concurrency.
- Add a Delay.
- Save the settings.
- Start one new analysis.
- Monitor firewall and origin logs for
429,403, timeout, or load errors.
Avoid changing Max URLs, Depth, JavaScript, Concurrency, and Delay at the same time. One change at a time makes the result understandable.
Configure search tracking
Set the refresh frequency for supported search-tracking features and confirm the project's country and language. Search settings do not control website crawling; diagnose crawling through access and crawler settings instead.
Configure availability alerts
Choose whether uptime monitoring is enabled and where notifications should be sent. Uptime checks and full website crawls are different workflows: an available homepage does not prove that every page is crawlable.
Troubleshoot unexpected crawl results
Check crawl depth, internal links, robots.txt, redirects, JavaScript navigation, and page responses. Max URLs raises capacity but does not create a discovery path or remove access blocks.
Allowlist 5.75.247.29 in rate-limit rules when possible. Otherwise lower Concurrency, add Delay, save, and run a new analysis while monitoring logs.
Disable it again if the real cause is firewall access, robots rules, server errors, or uncrawlable links. JavaScript rendering helps only when required content exists after client-side execution.
Select the new completed crawl, not the historical snapshot. Then trace one missing URL with the crawler troubleshooting guide.
Delete a project with care
Deleting a project removes it from normal workspace use. Confirm the exact project and retain any reports or records your team needs before deletion. If your only goal is to fix a crawl, keep the project history and resolve the underlying access or configuration issue instead.