The free Screaming Frog SEO Spider can find broken links, redirect detours, missing metadata, conflicting indexing signals and several other problems on a small website. These 14 checks turn that crawl into specific decisions about what to edit, what to investigate and what to leave alone.
The limit is 500 URLs per crawl, with several advanced features unavailable. A report that stops at the cap is a partial view, even if its issue lists look reassuring. Use the checks below for the URLs actually reached, and keep that coverage limit attached to your findings.
Set up a free crawl you can interpret
Download the SEO Spider from its official product page. Open it, use Spider mode, enter your preferred public homepage URL and click Start. Crawl a website you own or have permission to inspect.
Record the start URL, date, whether the crawl finished or hit its limit, and the important pages you expected to see. The allowance is expressed in URLs; it is not a guarantee that all of your important HTML pages will be reached. If a key page never appears, its condition remains unknown.
This guide follows official documentation and the free-versus-paid feature grid checked on November 21, 2026. It is not a hands-on crawl benchmark. The vendor's version 25 release notes list a 25.1 maintenance update dated November 4; use the current documentation if a label differs in your installation.
Free access does not include crawl configuration, JavaScript rendering, saving and reopening crawls, near-duplicate analysis, structured data validation, scheduling or the Google Analytics, Search Console and PageSpeed Insights integrations. The checks below use the default text-only crawl and manual exports.
| Question | Where to begin | Useful evidence |
|---|---|---|
| Are links leading to failures or detours? | Response Codes | Destination, status and linking page |
| Does the page describe itself accurately? | Page Titles, Meta Description, H1 and H2 | URL, current text and intended page purpose |
| Are access and indexing signals intentional? | Canonicals, Directives and Response Codes | Declared target or directive and affected URL |
| Do images and language versions need repair? | Images and Hreflang | Asset or alternate URL plus source-page context |
| Can visitors reach important pages easily? | Internal | Crawl depth and discovered inlinks |
A text-only crawl can miss content and links created by JavaScript. An empty report for such a site does not establish that it is healthy. For the broader sequence of crawl, rendering and Google-observed checks, use Screpy's technical SEO checklist.
1. Find broken internal links and their source pages
Open Response Codes, choose Client Error (4xx) and inspect the actual status code. Start with internal destinations returning 404 or 410, especially those linked from navigation, service pages or other important content.
Select a failed URL, then open Inlinks in the lower pane. The destination tells you what failed; the source page and anchor tell you where to make the edit. The official broken-link tutorial also documents Bulk Export > Response Codes > Client Error (4XX) Inlinks for a repair list.
For example, if a service page links to a retired booking URL, update that source link to the real current booking page. Restore a missing destination when the content should still exist. Use a redirect only when there is a relevant replacement; sending every deleted URL to the homepage can create a misleading journey.
Keep 403 and 429 results out of the “deleted page” bucket. They indicate access refusal or rate limiting and need a different investigation. A correct 404 for intentionally removed content is not itself a reason to recreate the page.
2. Separate server failures from access problems
In Response Codes, review Server Error (5XX) and No Response separately. A 500 or 503 is different from a connection timeout, and both differ from a 403 or 429.
For each important affected URL, record the exact result and time. Check whether the destination opens normally for a visitor, then ask the site administrator to compare the failure with server or access-control evidence. The Response Codes reference explains the categories.
A single crawler failure does not prove that all visitors or Googlebot receive the same response. Equally, a successful browser visit does not erase the recorded failure. Keep both observations until you understand the difference.
Do not solve a temporarily failing live page by deleting it or redirecting it elsewhere. Establish whether it is a server fault, an access restriction or an intermittent connection problem, fix that cause and repeat the check. If access prevents a trustworthy crawl, mark the affected URLs unverified.
3. Replace internal links that take redirect detours
Choose Response Codes > Redirection (3XX). Select a URL and inspect its Inlinks to find internal links still using an old address. Review Reports > Redirects > Redirect Chains when you need the sequence of hops or evidence of a loop.
Suppose an article links to an HTTP address, which redirects to HTTPS and then to a renamed page. Once you verify the final page is the intended destination, update the article's link to that final canonical URL. This removes unnecessary hops from that journey.
Keep the old redirect when it still serves external links, bookmarks or other valid entry points. Updating internal links and maintaining useful redirects are complementary jobs.
A 302 is not automatically wrong, and a redirect is not inherently an SEO failure. Check whether the move is temporary or permanent, whether the destination is relevant and whether the route actually resolves. A loop needs correction; a legitimate single redirect may need no change.
4. Review missing and duplicate page titles
Open Page Titles and inspect Missing and Duplicate. Start with pages you intend to publish and have appear in search, rather than rewriting every utility URL in the crawl.
Compare each flagged title with the page's actual subject. If two distinct service pages both say “Services | Example,” give each a specific, accurate title. If many pages share the same mistake, inspect the CMS template before editing them individually.
Duplicate titles do not prove the page bodies are duplicates. They can also reveal a naming problem or several URLs exposing the same page; check the destination and canonical before choosing the fix.
Length and pixel-width filters are editing aids. Google's title-link guidance does not set a fixed character limit and explains that Google can use several sources for a title link. Prioritize clarity and a truthful distinction between pages over making every title pass a crawler threshold.
5. Check meta descriptions against page intent
In Meta Description, use Missing and Duplicate to assemble a short review queue. Read the page before writing its description.
A pricing page needs a different summary from a support guide, even when both use the same product name. Describe what the visitor can actually learn or do on that URL. If a template copies one description across an entire section, fix the template or provide appropriate page-specific fields.
A missing description does not mean Google cannot index the page. Google primarily creates snippets from page content and may use the meta description when it better describes the result, as its snippet documentation explains.
Export the URL and existing description alongside your proposed replacement. That preserves the connection between the field you reviewed and the page someone will edit, without treating a length warning as a guaranteed search-result display rule.
6. Check whether headings explain the page
Review H1 > Missing, Duplicate and Multiple, then use the H2 tab to inspect the main sections. Open a flagged page and compare its heading structure with what a visitor is trying to accomplish.
For example, a local service page headed only “Welcome” may need a clearer main heading. Repeated H1s across unrelated pages may point to a template that substitutes the site name for the page topic.
The H1 reference says the default extraction records the first two H1s while the occurrence count can reveal more. Do not mistake those two text columns for a complete inventory of every heading.
Use flags to prompt a structural review. Multiple H1s are not, by themselves, proof of a Google penalty. Check whether the main subject and section hierarchy are understandable, and remember that headings injected by JavaScript may be absent from the free text-only crawl.
7. Check declared canonical destinations
In Canonicals, inspect Canonicalised, Multiple Conflicting and Non-Indexable Canonical. Read the declared destination and its available status rather than treating every non-self-referencing URL as broken.
If a product page declares a retired product as canonical, ask whether the pages really represent the same content. If the destination redirects or fails, identify the intended current URL before changing the template. When the target was not fetched within the crawl's coverage, verify it separately and leave the result unknown until you have evidence.
A missing canonical is also worth reviewing, especially where URL variants exist, but it does not prove that Google failed to index the page. A canonical pointing elsewhere can be intentional consolidation.
The crawler reports declarations and its own indexability assessment. It does not report Google's selected canonical. Google's canonicalization guidance explains how canonical signals are combined. Keep the intended preferred page, internal links and declared canonical consistent; do not use canonical tags as a substitute for an unrelated redirect or deletion decision.
8. Find unintended noindex directives
Open Directives > Noindex. For an affected page, inspect both the Meta Robots and X-Robots-Tag fields. The exclusion may come from HTML, a server response header or both.
Compare the result with the page's intended role. A public service page may have inherited a staging-site exclusion. An account page or internal search result may correctly be excluded. The fix is to align the directive with that intention, not to remove noindex everywhere.
If you remove a noindex meta tag but the response header still supplies noindex, you have not resolved the whole finding. Record which source produced the directive so the right person can change it.
Google needs crawl access to see a noindex instruction. Blocking the same URL in robots.txt can prevent that instruction from being read. After a change, verify the live HTML and headers and repeat the crawl check. An “Indexable” crawler label still does not prove the URL is in Google's index.
9. Inspect URLs blocked by robots.txt
Choose Response Codes > Blocked by Robots.txt and review important pages and resources in the result. Compare each path with the site's live robots.txt and its intended access policy.
A rule blocking a private area can be correct. A rule that also catches public product or article URLs deserves investigation. Keep the URL and the relevant rule together in the repair record, and check which crawler group the rule applies to before recommending an edit.
This result describes the SEO Spider's crawl. It is not a universal test of every Google or AI crawler's permissions, and a blocked response leaves the page's content uninspected.
Keep fetching permission separate from indexing exclusion: robots.txt can stop crawling without reliably removing a URL from search. If the actual objective is to exclude a page from Google's index, use the noindex conditions in the previous check. Do not bypass a block or change crawler identity merely to produce a cleaner report.
10. Compare pages with identical HTML
Open Content > Exact Duplicates and inspect groups with matching Hash values. Screaming Frog's duplicate-content documentation distinguishes this free check from paid near-duplicate analysis.
The exact check compares the full HTML using an MD5 hash. It does not answer whether two differently marked-up pages tell substantially the same story. Small HTML differences can stop a match even when the visible text is nearly identical.
For a matching pair, compare the URLs, page purpose, canonical declarations and internal links. A tracking-parameter variant may need consolidation and cleaner internal links. Two URLs intentionally serving different audiences may need distinct content instead. Do not redirect or delete a page solely because a filter flagged it.
Record the pair and proposed preferred URL where appropriate. Absence from this list means no exact match was found within the captured set; it is not evidence that the entire website contains no duplicated or repetitive content.
11. Review image alt text in its page context
In Images, inspect Missing Alt Attribute and Missing Alt Text separately. Select an image and open Image Details to see the pages using it and the alternative text associated with each use.
An absent attribute and an intentionally empty alt="" are different conditions. W3C's guidance on decorative images explains when an empty alternative is appropriate. Do not fill every blank with the filename or a target keyword.
For an informative image, write an alternative that communicates its purpose on that page. If the same asset appears as meaningful content on one page and decoration on another, review the uses separately rather than making one global assumption.
The official image-alt tutorial documents Bulk Export > Images > Images Missing Alt Attribute & Text to retain source-page context. Keep that relationship in the handoff. Images introduced only after JavaScript execution may be missing from the free crawl, so the exported list is not a complete accessibility audit.
12. Triage large image files
Choose Images > Over 100kb and inspect file sizes, then use Image Details to find the source pages. Treat the filter as a queue for investigation, not a requirement that every image must be smaller than 100 KB.
A large hero photograph and a detailed downloadable diagram have different jobs. Check the image's displayed size, necessary detail, format and placement before compressing or replacing it. For a small card image loading a much larger source file, a correctly sized variant may be the useful fix.
The Images reference notes that exported size is in bytes; divide by 1,024 when comparing a spreadsheet value with kilobytes.
File size alone does not identify the Largest Contentful Paint element or measure Core Web Vitals. Those need separate performance evidence. The PageSpeed Insights integration is paid, so it is not part of this free workflow. After an image change, confirm both visual usefulness and the actual asset being loaded.
13. Check language alternates and return links
For a multilingual or multi-region website, open Hreflang and inspect the extracted language codes and alternate URLs. A single-language site without alternate versions can skip this check.
There is an important free-version boundary: the default crawl extracts hreflang annotations but does not follow their targets for discovery. Enabling “Crawl Hreflang” is a configuration change outside this free workflow. An annotation in the table therefore does not establish that its destination was fetched.
Start with one set of equivalent pages. Check whether the alternate was also discovered through ordinary links in your crawl. Where both pages were fetched, compare their annotations, available status codes and declared canonicals. For a counterpart outside that set, open its public source separately to verify the relationship; record it as unverified until then.
Look for invalid language or region codes, the wrong counterpart, a failing target or a missing reciprocal annotation. The Incorrect Language & Region Codes filter can help with syntax; interpret target-status and return-link filters only when the relevant counterpart has actually been checked. Google's localized-version guidance explains supported codes and reciprocal annotations.
If an English product page points to a French category page, a valid code does not make them equivalent. Keep that editorial check alongside the technical evidence. Empty filters in an incomplete crawl do not establish a complete cross-domain or sitemap-based language audit.
14. Find important pages buried in the crawl
In Internal, select HTML and sort Crawl Depth. Compare important pages with their discovered Unique Inlinks, then inspect the source pages behind those links.
Depth is measured from the URL where you started, not automatically from every possible visitor entry point. The Internal tab documentation also notes that redirects count as a level in this calculation.
If a useful service page is reached only through several old articles, consider adding a relevant link from the current services hub. The goal is a clearer route for visitors, not a mechanical rule that every page must sit within three clicks.
A partial crawl can undercount inlinks, and the discovered graph does not establish that an absent URL is an orphan. Keep this check to pages and relationships actually captured. For a page you expected but cannot find, mark discovery unresolved and investigate its navigation or rendering separately instead of assigning it a misleading depth or a zero-link result.
Export a repair list and verify the changes
Use the Export button for the filtered tab you are reviewing. For problems involving links or image uses, choose the relevant source-page export so the person making the edit can find the actual occurrence.
Keep the start URL, crawl date, completion or cap status, and known coverage gaps beside the report. Exporting a spreadsheet is available as a manual handoff; it is different from saving and reopening a crawl, which requires a licence.
Copy this CSV into your working sheet. These are fictional, unchecked examples, not crawl results. Replace the example addresses and leave observations blank until you verify them.
check,source_url,target_url,observed_result,intended_state,proposed_action,owner,recheck_status
internal_link,https://example.invalid/services/,https://example.invalid/old-booking/,,working booking destination,verify target and update source link,,not_checked
noindex,https://example.invalid/services/,,,public page eligible for indexing,inspect HTML and response headers,,not_checked
image_alt,https://example.invalid/services/,https://example.invalid/images/team.jpg,,appropriate alternative for this use,review image purpose on source page,,not_checked
Group confirmed findings by cause: a shared template can explain many repeated rows. Prioritize unintended exclusions and broken important journeys before cosmetic length warnings.
After the CMS or template is fixed, start a fresh crawl and compare the relevant exported rows manually. Keep the same start URL and record any coverage difference. A lower issue count is not progress if the second crawl simply missed the affected pages.
If the 500-URL cap or a missing feature prevents the job, the next decision is about suitable coverage, not another clean-looking partial report. Screpy's free Screaming Frog alternatives and crawl limits compares other options for that separate decision.