Compare Website Crawls With MCP

To compare website crawls with MCP, select two completed snapshots, request their metric differences, and inspect the same URLs in both runs before interpreting the changes. The comparison can reveal what needs investigation after a release. It cannot tell you, from totals alone, which fixes worked or why Google traffic changed.

We used Screpy MCP to compare two crawls of Screpy’s own website, completed on October 2 and October 9, 2026. We retrieved the saved results on October 11. The health score stayed at 81, even though the later crawl contained 90 more pages, nine fewer critical issues in the aggregate, and more warnings.

That mixed result is the useful part of the example. We will show the actual comparison arguments, explain the returned numbers, and check one matching page in both snapshots. The result is a repeatable review process that distinguishes measured changes from explanations that still need evidence.

Choose a baseline and a comparison crawl

Authorize a compatible client using the Screpy MCP setup guide, list the accessible projects, and select the website you intend to inspect. Then list its saved crawls. Choose completed runs and record their identifiers, rather than asking the assistant to use an ambiguous “latest” result.

Our pair was chronological: October 2 first, October 9 second. We did not establish which deployments occurred between them. For a release review on your own site, a useful baseline is a completed crawl before the change, paired with a completed crawl after the release. That timing still needs to be recorded, not assumed.

Create a comparison record before analyzing the totals:

Both of our responses returned a maximum-page value of 5,000. Their progress objects also reported a current depth of 10; that field should not be relabeled as a configured depth limit. We did not verify all seeds, exclusions, or rendering settings. Those remain limits on the comparison.

The SEO MCP overview explains the tool connection. For this task, the important contract is narrower: keep the pair fixed, record what is comparable, and flag unknown settings. If configurations differ, you can still examine the results, but the report must describe those differences before attributing a changed total to the website.

Ask MCP for the comparison

Use compare_crawls with explicit first and second selections. These are the verified argument names; replace the placeholders with identifiers from your own accessible project and completed runs:

{
  "first": {
    "project_uid": "YOUR_PROJECT_UID",
    "crawl_uid": "YOUR_BASELINE_CRAWL_UID"
  },
  "second": {
    "project_uid": "YOUR_PROJECT_UID",
    "crawl_uid": "YOUR_COMPARISON_CRAWL_UID"
  }
}

Our result included both crawl summaries and a data.deltas object. The following is the observed deltas subset from our October 2 to October 9 comparison:

{
  "health_score": 0,
  "pages_total": 90,
  "critical_issues": -9,
  "warnings": 3954,
  "notices": 2262
}

The direction is second minus first. In this pair, pages_total: 90 means the later total exceeded the earlier total by 90. Reversing the pair would reverse the direction of the comparison. Include that direction in the report so a reader does not interpret a negative count as deterioration merely because it has a minus sign.

Save the full response, not just a paraphrased assistant summary. The individual snapshots contain the dates and values needed to check each subtraction. The Screpy MCP tool reference also describes the page and source tools available for further investigation.

This particular comparison response contained aggregate differences. It did not supply an automatic HTML diff for every page, identify the owners of issues, or explain the cause of a change. An assistant can calculate and organize additional information from retrieved data, but that derived analysis needs to be labeled separately.

Before moving on, confirm that both returned crawl identifiers match the requested pair and both runs are complete. A correct-looking table built from the wrong snapshots is still the wrong comparison.

Read changes without losing the denominator

Start with the original values alongside each difference. These measurements came from the two completed Screpy snapshots; they are not a simulated performance test:

The unchanged score does not mean nothing changed. The critical count fell, the crawl included more pages, and the warning and notice totals rose. A single score compresses several signals; use it as an overview, then examine the fields relevant to the work you are reviewing.

The page total changed too. Repeated template findings can contribute more occurrences when a crawler discovers more URLs. As a simple diagnostic, we divided warning counts by each response’s total-page count:

October 2: 44,461 ÷ 1,264 ≈ 35.2 warning occurrences per crawled page.
October 9: 48,415 ÷ 1,354 ≈ 35.8 warning occurrences per crawled page.

These ratios are our calculations. They are not percentages of affected pages, a Screpy score, or evidence that every page has that many warnings. They help prevent an increase in raw totals from being interpreted without considering crawl size. Comparable field definitions still matter.

Likewise, 90 more pages is a net total difference, not proof that exactly 90 new pages were published. Some URLs could have entered the crawl while others disappeared. The count alone cannot identify either set.

Nine fewer critical issues is also a net difference. It does not establish that nine identified problems were repaired. Affected URLs might have been corrected, excluded, redirected, or omitted from the later discovery path. You need page-level evidence to distinguish those explanations.

Screpy Website Audit provides the underlying crawl findings. Treat the comparison as a shortlist of questions: what changed, on which URLs, under what coverage, and whether the change was intended. That produces a more useful release review than labeling the whole site “better” or “worse” from one number.

Investigate the same pages in both snapshots

When the question depends on which URLs changed, retrieve the page lists for both snapshots and follow their continuation cursors. Compare the resulting URL sets: URLs present in both, URLs found only in the later crawl, and URLs found only in the earlier crawl. The last group means “not found in the later saved crawl,” not automatically “deleted from the website.”

For retained URLs, inspect matching fields in each snapshot. We queried get_page_details for https://screpy.com/ using each of our two crawl identifiers. The argument shape is:

{
  "project_uid": "YOUR_PROJECT_UID",
  "crawl_uid": "ONE_OF_YOUR_SELECTED_CRAWL_UIDS",
  "url": "https://screpy.com/"
}

Repeat the request for the other selected crawl while keeping the URL fixed. Our saved homepage details showed:

The title remained “AI SEO Analyzer & Website Audit Tool | Screpy,” and the declared canonical matched the homepage URL in both responses. We did not review full URL sets or a historical text diff for this example; it demonstrates one matching-page comparison.

The reduced word count does not prove content quality declined. The additional internal links do not establish improved architecture. One fewer JSON-LD block does not prove valid markup was lost. Inspect the actual content, links, and markup before deciding whether those changes are expected or problematic.

For a migration, the same logical page may have a different URL. Keep an explicit old-to-new mapping and document why the pages are equivalent. Do not silently remove query parameters or collapse paths that could represent different pages. Screaming Frog’s crawl-comparison guide describes its own URL Mapping feature; the Screpy aggregate MCP response we used does not provide that feature automatically.

To validate a specific fix, compare the affected URLs that occur in both runs first, then review new and missing URLs separately. This reduces the risk of celebrating a lower issue total when the crawler simply stopped seeing the pages that had the issue.

Turn changes into a release review

A comparison report should tell the person responsible for the website what to check next. Keep observations, unanswered questions, and proposed actions in separate columns. This example uses our measured pair, but the investigations are recommendations, not work we completed:

For a real release, attach the deployment date and the intended changes. Ask the developer or editor to connect each change to the relevant URL or component. If that record is unavailable, report the missing information. Our two snapshots show differences; they do not identify an intervening release or its cause.

You can give an assistant a constrained report request after retrieving the evidence:

Use only the retrieved first and second crawl responses and the
matching page details. State the dates and comparison direction.
Separate returned metrics from calculations. For each material
change, list the supporting field, unresolved explanation, next
check, and proposed owner. Do not label a net decrease as a fixed
issue without matching affected-URL evidence.

The request helps organize the report; it does not replace tool calls or supply missing data. Review the resulting statements against the saved responses. A convincing explanation is still a hypothesis if the field-level evidence does not support it.

Assign each material investigation to someone who can verify it. When a correction is needed, specify what the live page should show and what a subsequent completed crawl should record. Our crawler troubleshooting guide explains the discovery context behind those saved findings.

Keep the report with its source responses and validation outcome. That record lets the next reviewer see whether a change was expected, investigated, corrected, or still unresolved without reconstructing the assistant conversation.

Keep crawl changes separate from search results

A crawl comparison describes what the crawler recorded at two times. Search-performance data describes a different observation. Neither an unchanged audit score nor a lower issue count establishes what happened to impressions, clicks, or rankings.

If the review includes a performance question, record the relevant release and crawl dates alongside the search-data period. Check that the pages being discussed are the same pages in the performance report. A sitewide total can hide differences between page groups, just as an aggregate crawl count can.

Treat any proposed explanation as something to test. Content changes, internal links, seasonal demand, and other releases may all be relevant to a traffic change. Our example did not measure those factors, so it cannot establish a ranking benefit or loss from the observed homepage differences.

Before sharing a comparison report, confirm:

• Both selected snapshots are complete, and their identifiers are saved.
• The baseline and comparison direction are stated explicitly.
• Known configuration differences and unknown settings are recorded.
• Counts retain their units, and calculations are labeled.
• Net differences are not presented as lists of new pages or completed repairs.
• Important conclusions point to matching URL evidence.
• Proposed actions have an owner and a live or fresh-crawl validation step.

For questions beyond the pair, use the technical SEO checklist to identify the additional checks needed. Avoid extending a narrow comparison into a full audit without gathering the necessary evidence.

To compare website crawls with MCP reliably, keep the selection reproducible and the interpretation tied to the fields you actually retrieved. The benefit is a faster path from a changed total to a specific question someone can investigate. That is enough to make the comparison valuable without claiming that the numbers alone explain the website’s SEO performance.