Crawl your website
Configure, run, and verify a Screpy crawl for a reliable website audit.
Screpy starts with a crawl because it is the shared data source for Pages, Links, Images, on-page findings, and Quick Wins. Treat a completed crawl as a snapshot: it describes what Screpy could reach and analyze at that point in time.
Before you run a crawl
Use the root domain for the project and make sure the website is publicly reachable. If a firewall, CDN, or WAF blocks automated traffic, allow the crawler before starting. robots.txt also needs to permit ScrepyBot; a blocked crawl cannot produce reliable audit results.
Choose a URL limit and depth that match the question you are trying to answer. A smaller scope is useful when checking a section after a release. A broader crawl is better for finding site-wide technical issues. More depth and more URLs give a more complete picture, but also use more of the crawl allowance available to the project.
| Control | What it changes | Use it when |
|---|---|---|
| Maximum URLs | The number of discovered pages that Screpy can include in the crawl | Your site is larger than the current audit, or you need wider coverage. |
| Crawl depth | How many link levels Screpy follows from the start page | Important content is deeper than the main navigation. |
| Crawler access | Whether the project can run website audits | You need to pause or restore crawling for a project. |
Set these controls in Project settings before the next crawl. A completed crawl keeps its own scope; changing settings does not rewrite a past snapshot.
Start and follow a crawl
The project setup starts the first crawl automatically. To refresh the audit later, open Pages, Links, or Images, choose the crawl context shown above the report, and use Analyze when a new manual crawl is available.
- Confirm the domain, URL capacity, and depth are appropriate.
- Start the crawl and leave the project accessible while Screpy analyzes it.
- Wait for a completed snapshot before treating counts as final.
- Open Quick Wins for priorities, then use the page, link, and image reports to investigate the affected URLs.
Manual starts are intentionally limited: after one successful manual start, wait up to one hour before starting another. Screpy also prevents a second crawl from running while the current one is in progress.
Read the crawl status first
Open the project and wait for the crawl to complete before treating its findings as final. While a crawl is running, some lists and counts can be incomplete. When it finishes, start with the dashboard and Quick Wins, then open Pages, Links, and Images only when you need to investigate a specific issue.
If the crawl finds fewer URLs than expected, check the start domain, redirects, robots rules, login walls, and blocked resources. A low count does not automatically mean Screpy missed pages; it can also mean the site does not expose crawlable links to them.
When should you run another crawl?
Run a new crawl after a meaningful technical change: a migration, redirect rollout, template change, indexing fix, navigation change, or a large content release. Avoid comparing an in-progress crawl with a completed one. Compare completed snapshots so you can separate a real improvement from a partial data set.
You do not need a new crawl for every editorial tweak. Prioritize changes that affect what a crawler can reach, render, or interpret, then use the follow-up crawl as the verification step.
If a crawl does not look right
| Symptom | Check first |
|---|---|
| Fewer pages than expected | Start domain, URL capacity, crawl depth, internal links, redirects, robots.txt, and login walls. |
| Crawl cannot reach the site | Public reachability and firewall, CDN, or WAF rules for the Screpy access details shown during setup. |
| A new crawl cannot start | Whether another crawl is running or the one-hour manual-start window is still active. |
| Audit data is unavailable | Whether crawling is enabled and included in the current project plan. |
For an overview of the product workflow, see the Screpy Website Audit page.
For the data scope and limitations behind this guide, see how Screpy measures website-audit data. For a primary reference on crawler access rules, see Google's robots.txt guide.
Use one crawl as the baseline
Record the date and scope of the crawl you use as a baseline. This makes later changes in error counts or page coverage easier to explain.