Crawling your website with Screpy
Understand how Screpy discovers URLs, configure crawl depth and JavaScript rendering, verify coverage, and diagnose missing website pages.
The Screpy SEO Crawler creates the source data for Pages, Links, Images, on-page findings, and Quick Wins. Screpy starts from the project URL, follows eligible links, and analyzes accessible responses within the project's crawl settings.
Keep crawler access enabled
Allow 5.75.247.29 through your firewall, WAF, CDN, host, and security plugins. Keep ScrepyBot allowed in robots.txt. Use the crawler access guide whenever infrastructure changes.
How Screpy discovers website pages
Screpy can discover a URL when it appears in crawlable website navigation and remains within the configured project scope. Coverage is affected by:
- internal
<a href>links and the distance from the starting page; - maximum URL capacity and crawl depth;
- redirects, hostname changes, and canonical structure;
robots.txtand page accessibility;- JavaScript-only links or content;
- duplicate, parameterized, and non-HTML URLs;
- server errors, timeouts, and rate limits.
A sitemap describes URLs, but it does not make blocked or unreachable pages crawlable. Important pages should also be connected through normal internal links.
Configure crawl scope deliberately
| Setting | Use it for | Do not use it for |
|---|---|---|
| Max URLs | Expanding a successful crawl that reached its capacity. | Fixing firewall, robots, timeout, or discovery problems. |
| Depth | Reaching pages several link levels from the start page. | Finding orphan pages with no crawlable path. |
| JavaScript | Rendering content and links created only in the browser. | Bypassing bot protection or fixing server errors. |
| Concurrency | Balancing crawl speed and server capacity. | Overriding a hard request-rate rule. |
| Delay | Slowing requests for sensitive servers. | Repairing inaccessible URLs or navigation. |
Advanced settings live in Project settings. Save one intentional change at a time and evaluate it in the next completed crawl.
Start and follow a crawl
The first crawl starts during project setup. For a later snapshot, open Pages, Links, or Images, choose the crawl context above the report, and use Analyze when available.
- Confirm crawler access, domain, Max URLs, and Depth.
- Start one analysis and follow its status.
- Wait for completion before treating counts as final.
- Compare expected sections with the crawled URL inventory.
- Open Quick Wins and inspect the affected records before changing the site.
A successful manual crawl start has a 10-minute interval, and the same project cannot run a second crawler analysis while one is active.
Diagnose a missing page
Choose one important missing URL and answer these questions:
- Does it return usable HTML without authentication or a bot challenge?
- Does
robots.txtallow its path forScrepyBot? - Is it linked from a crawled page with a normal anchor link?
- Is that path within the selected Depth?
- Did the crawl already reach Max URLs?
- Does the link redirect to another host or loop?
- Does the link exist only after JavaScript runs?
- Do server logs show
403,429,5xx, or timeout behavior?
This single-URL trace usually identifies the cause faster than increasing every crawl setting.
Interpret crawl status before results
| Crawl state | How to use its data |
|---|---|
| Queued or running | Progress only; totals can still change. |
| Completed | Stable snapshot for review and comparison. |
| Failed or blocked | Diagnostic evidence; fix the cause before using its totals. |
| Rate-limited | Reduce request pressure or allowlist the crawler, then rerun. |
Technical crawling questions
A page with no crawlable internal path may not be discovered during normal traversal. Link important pages from relevant website sections and ensure those links are accessible within the crawl depth.
Links implemented only as click handlers or created after scripts run may not exist in initial HTML. Prefer real anchor links for crawlability. Enable JavaScript rendering only when the site genuinely requires it.
Compare the count with Max URLs. If they match and access is healthy, increase capacity within the plan. If they do not match, check discovery, depth, blocks, timeouts, and rate limits first.
External destinations can be checked as links but are outside the project's website page scope. They may also block or rate-limit automated checks independently from your own site.
For symptom-based fixes, use Fix Screpy crawl and website analysis problems. For your first run, start with Run your first Screpy website crawl.