Getting started with Screpy
Prepare your website for Screpy, create your first project, run a complete crawl, and fix common website analysis problems.
Screpy turns a website crawl into practical SEO work. The Project Workspace keeps website settings, crawl history, and ongoing analysis connected. A reliable first analysis starts before you create the project: the crawler must be able to reach your real pages without being stopped by a firewall, bot challenge, rate limit, or robots.txt rule.
Allow the Screpy crawler before you add the project
Add 5.75.247.29 to the allowlist in every firewall, WAF, CDN, hosting security tool, and security plugin protecting your website. Also allow the ScrepyBot user-agent in robots.txt.
Set up your first website analysis
Check whether your website is ready
Confirm that the public homepage loads, HTTPS works, and visitors do not have to pass a login, CAPTCHA, maintenance screen, or country restriction. Use the website analysis checklist before changing crawl limits.
Allow Screpy through your security layers
Allow the crawler IP in every layer that can reject automated traffic. The crawler IP and firewall guide includes verification steps and fixes for common WAF and bot-protection problems.
Check robots.txt
Open https://yourdomain.com/robots.txt and make sure it does not block ScrepyBot or the sections you want analyzed. Follow the robots.txt guide for safe examples and conflict checks.
Create the project
Add the final public version of your website address and choose a crawl scope that fits the site. See how to add a Screpy project.
Wait for a completed crawl
The first crawl starts after setup. Counts may change while it is running, so use only a completed crawl as your SEO baseline. See how to run your first crawl.
Choose the guide that matches your problem
Website analysis checklist
Check access, redirects, security, robots.txt, and rate limits before setup.
Allow the Screpy crawler
Allow the crawler IP and diagnose firewall, WAF, CAPTCHA, and bot protection.
Configure robots.txt
Use safe ScrepyBot rules and fix robots.txt blocks.
Add a project
Choose the correct website address and complete project setup.
Run your first crawl
Understand crawl settings, progress, limits, and completed results.
Understand the dashboard
Find each Screpy feature and choose the right report for your next question.
Fix crawl problems
Troubleshoot failed, blocked, incomplete, or one-page website analyses.
What happens after the first crawl?
Open Quick Wins for prioritized opportunities. Then inspect the affected pages, links, and images before changing the website.
Keep the same project as your long-term reference point. Future crawls, search visibility, performance monitoring, and reports use its history to show what changed.