Starting URL and crawl scope
Begin from the project website and follow accessible internal links within the selected crawl capacity and depth.
- Project-level crawl scope
- Internal URL discovery
- Completed crawl inventory
SEO Crawler Tool for Technical Site Analysis
Run a cloud crawl and map the website from its internal links instead of checking URLs one by one. Configure crawl capacity, depth, JavaScript rendering, concurrency, and request delay; then inspect the exact Pages, Links, and Images records behind every technical pattern.
TL;DR
Too Long; Didn't Read
Show summary Hide summary
TL;DR
Too Long; Didn't Read
What Does an SEO Crawler Do?
Screpy SEO Crawler starts from the project website, follows accessible internal links within the selected scope, and creates Pages, Links, and Images inventories. Teams can configure discovery and request behavior, inspect the exact records behind technical patterns, export filtered data, and create comparable crawl snapshots after a website change.
Scope and access
- Screpy follows accessible internal links from the project website within the configured URL capacity and crawl depth.
- Screpy records crawl status and access context so a blocked response is not mistaken for an on-page SEO defect.
Crawl controls
- Screpy supports configurable URL capacity, link depth, and optional JavaScript rendering for client-side content.
- Screpy supports concurrency and request-delay controls when a website requires slower crawling.
Technical inventories
- Screpy Pages records URL responses, depth, redirects, canonicals, metadata, headings, indexability, and related page signals.
- Screpy Links and Images connect destinations and assets with their status, attributes, usage, and crawled source pages.
Verification and exports
- Screpy preserves each completed crawl as a dated snapshot that can be compared with a later crawl using equivalent scope.
- Screpy filters and CSV or Excel exports turn crawl records into a focused implementation and verification queue.
Control what the SEO crawler can discover.
A useful crawl starts with explicit scope and access. Screpy lets teams adapt discovery, rendering, request pace, reports, and repeat analysis to the website being investigated.
Maximum URL capacity
Set a crawl capacity that fits the website and plan, then check whether the completed crawl reached that limit before interpreting coverage.
- Plan-aware URL capacity
- Coverage limit context
- Coverage interpreted against the selected limit
Control how many internal link levels Screpy follows from the starting page and identify valuable URLs buried deep in the architecture.
- Depth recorded for crawled pages
- Deep-page investigation
- Architecture review
JavaScript rendering
Render client-side content when important links or page elements do not exist in the initial HTML response.
- Optional rendered-page crawling
- Client-side link discovery
- Rendered content analysis
Concurrency and request delay
Adjust crawl speed for the website instead of forcing the same request pattern on every server or application.
- Concurrent request control
- Configurable request delay
- Rate-limit-aware crawling
Distinguish an SEO finding from a blocked crawl by checking robots rules, firewalls, bot protection, response status, and returned HTML.
- ScrepyBot access checks
- robots.txt review
- Blocked or challenged response context
Filterable crawl reports
Move from a site-wide total to the exact pages, links, and images behind a pattern, then export the filtered records for implementation.
- Pages, Links, and Images inventories
- Issue and record filters
- CSV and Excel exports
Dated crawl snapshots
Preserve each completed crawl as a snapshot, repeat the analysis with comparable settings, and verify what changed after deployment.
- Historical crawl selection
- Before-and-after comparison
- Recurring technical review
Inspect every discovered URL as a crawl record.
Turn a completed crawl into a filterable URL inventory. Review response status, final destination, crawl depth, response time, metadata, headings, canonical, indexability, content, structured data, social tags, and related page signals without opening each URL manually.
Learn more about pages report- Filter the full inventory or focus on one technical pattern
- Compare representative URLs across templates and sections
- Export the exact page records needed for implementation
Map destinations back to every source page.
Review internal and external destinations with their response status, anchor text, attributes, usage count, and crawled source pages. A broken or redirected URL becomes a specific navigation, content, template, or component fix.
Learn more about links report- Find failed and redirected destinations in one inventory
- Trace each destination to the pages that reference it
- Separate internal architecture from external references
Review image delivery together with page usage.
Group crawled image sources and inspect response status, alt text, dimensions, responsive attributes, loading behavior, source type, and every page that uses an asset before changing it.
Learn more about images report- Find unavailable, inaccessible, and incomplete image records
- Review alt text, dimensions, srcset, sizes, and loading
- Trace repeated assets back to their crawled source pages
Create a baseline, change the website, and crawl again.
Treat each completed crawl as a dated snapshot. After a release, migration, navigation change, or technical fix, repeat the analysis with comparable settings and verify the affected URL pattern instead of relying on one changing total.
Learn more about crawl snapshots- Select historical crawls without changing completed results
- Compare equivalent scope before and after deployment
- Separate real website change from different crawl settings
Turn a crawl into evidence you can verify.
Keep the scope visible, inspect the records behind a pattern, fix the underlying implementation, and use the next crawl to confirm the result.
- 01
Define the crawl before interpreting it
Confirm the starting URL, URL capacity, link depth, JavaScript mode, concurrency, request delay, robots rules, and crawler access. These settings determine what the snapshot can represent.
- 02
Let internal links reveal the site structure
Screpy follows accessible internal paths and records each discovered URL, response, depth, destination, link relationship, and related page or image data within the configured scope.
- 03
Filter the exact records behind a pattern
Use Pages, Links, and Images reports to move from a total to representative URLs, source pages, redirect behavior, metadata, canonicals, or image usage before choosing a fix.
- 04
Publish the fix and create a comparable snapshot
Run another crawl after the website changes. Keep scope and settings comparable, then verify that the intended URLs and underlying pattern changed for the right reason.
Use the crawler before and after structural website changes.
Crawl data is most useful when the team still knows what changed. Create a baseline before high-risk work and a comparable snapshot after it ships.
Website migrations and URL changes
Record important URLs, redirects, final destinations, canonicals, and internal paths before launch, then crawl the new structure with the same scope.
JavaScript websites and applications
Use JavaScript rendering when navigation or meaningful page content is created in the browser, then confirm those URLs appear in the crawl inventory.
Internal linking and site architecture
Review crawl depth, source pages, anchors, attributes, and repeated navigation paths to find important content that is weakly connected or difficult to discover.
Release and template monitoring
Recrawl after routing, navigation, CMS, template, and component changes to catch repeated technical regressions while the implementation context is still fresh.
Compare Screpy with Screaming Frog when deployment model, crawl controls, custom extraction, recurring monitoring, team access, and reporting affect the decision.
Choose raw crawl data or a prioritized website audit
Use SEO Crawler to map discovery paths and inspect page, link, and image records. Use Website Audit when the team needs grouped findings, issue prioritization, and a broader technical health workflow.
Configure a cloud crawl, map reachable URLs, and inspect the raw Pages, Links, and Images records behind technical patterns.
Turn a completed crawl into grouped findings, affected URL patterns, priorities, exports, and a repeatable fix-verification workflow.
Apply crawl evidence to the website model you operate.
Shared templates and changing URL sets create different risks for SaaS acquisition sites and ecommerce catalogues.
Review landing pages, documentation, shared templates, and release regressions as one acquisition surface.
Audit category and product page types, broken shopping paths, and structural catalogue changes.
Screpy documentation
Learn how to crawl your website
Use the Screpy guides to configure crawl scope, allow crawler access, interpret Pages, Links, and Images reports, and diagnose incomplete results.
Related SEO guides
Learn the concepts behind this workflow and apply them with practical SEO guidance.
SEO
How Does an SEO Crawler Find Technical Website Issues?
An SEO crawler follows links and checks URLs for broken links, redirect chains, duplicate pages, canonical errors, indexability issues to prioritize fixes.
Read article SEO
What SEO Alerts Should You Monitor Before Traffic Drops
SEO alerts to track in GA4 and Search Console: indexing and crawl errors, ranking volatility, Core Web Vitals shifts, plus security or manual actions.
Read article SEO
How to Optimize XML Sitemaps for Better Indexing
Optimize XML sitemaps with canonical-only URLs, accurate lastmod dates, and sitemap index files, then validate in Search Console to spot common crawl errors.
Read article SEO
What Is an SEO Report Generator and How Does It Work?
An SEO report generator gathers rankings, traffic, backlinks, and audit data into branded, scheduled reports that highlight progress, issues, and next steps.
Read articleSEO Crawler FAQ
Understand crawl discovery, scope, JavaScript rendering, access, request pace, reports, snapshots, and the difference between crawl data and a website audit.
What is an SEO crawler?
An SEO crawler starts from a website URL, follows accessible internal links within a defined scope, and records technical information about the pages and resources it reaches. The resulting inventory helps teams investigate discovery paths, response codes, redirects, canonicals, internal links, metadata, images, and repeated site patterns.
What is the difference between an SEO crawler and a website audit?
The SEO crawler creates the technical inventory: discovered URLs, responses, paths, page data, links, images, and crawl context. Website Audit interprets that crawl as grouped findings, affected patterns, priorities, and a fix-verification workflow. Use the crawler when you need discovery and raw records; use the audit when you need a broader health review and prioritized action.
How does Screpy discover website pages?
Screpy begins from the project website and follows accessible internal links within the configured URL capacity and crawl depth. A page may be missing when it is not linked, sits beyond the selected depth, is blocked, redirects outside the project host, depends on JavaScript rendering, or returns unusable content.
Can Screpy crawl JavaScript websites?
Screpy provides an optional JavaScript setting for websites whose important links or content are created in the browser. Rendering can improve discovery for those sites, but it does not bypass robots rules, authentication, firewalls, bot challenges, timeouts, or the configured crawl scope.
Can Screpy crawl a protected or blocked website?
The crawler must receive the real page without a login, CAPTCHA, browser challenge, firewall block, or rate-limit rejection. Allow ScrepyBot and the documented crawler IP through each security layer, confirm robots.txt access, and verify that a successful response contains the real page HTML.
What do maximum URLs and crawl depth control?
Maximum URLs caps how many discovered URLs the crawl can store. Crawl depth limits how many internal link levels Screpy follows from the starting page. Increasing either setting cannot fix blocked access, missing internal links, redirects, rate limits, or unusable responses.
What is included in the Pages, Links, and Images reports?
Pages records crawled URLs and their response, depth, metadata, headings, canonical, indexability, content, structured data, social, and related signals. Links connects destinations with status, anchors, attributes, usage, and source pages. Images groups image sources with response, alt text, dimensions, delivery attributes, and page usage.
Can the crawler find broken links and redirect paths?
Yes. The Links report helps identify failed or redirected destinations and trace each one back to the crawled pages that reference it. This gives developers and editors a specific source page, shared component, or template to fix instead of only a destination URL.
Can I control how quickly Screpy crawls my website?
Yes. Concurrency controls how many requests can run at once, and delay adds time between requests. Reduce concurrency or increase delay when the host rate-limits automated traffic or struggles with request bursts.
Can I compare crawl results over time?
Yes. Each completed crawl is a dated snapshot. Run a new crawl after a release or fix and compare snapshots with equivalent scope and settings so changes in coverage or issue totals are not confused with a different crawl configuration.
Do I need an SEO crawler if I already use Google Search Console?
They answer different questions. Search Console reports Google Search performance and selected indexing information. An SEO crawler proactively follows your internal links and builds a technical inventory you can inspect before Google reports a problem or traffic changes.
Does fixing every crawler finding guarantee higher rankings?
No. Crawl data helps teams discover and verify technical conditions, but rankings also depend on relevance, content quality, authority, competition, demand, indexing, and how search engines interpret each page. Treat crawler filters as evidence to investigate, not automatic ranking instructions.