Crawl Traps

Crawl Trap Candidates highlights paths with unusually high request volume and many query-string variations. These patterns can create a very large or effectively unlimited URL space for crawlers without adding distinct search value.

Common sources include faceted filters, sort orders, calendars, internal search, tracking parameters, session IDs, pagination errors, and combinations of optional parameters.

Read the signals

High traffic alone is not a trap. A large product catalog can legitimately receive many crawler requests. The concern is high, repeated expansion into URLs with little unique value.

Recognize common trap patterns

Faceted navigation

Filters such as color, size, price, brand, availability, and sort order can multiply into thousands of combinations. Some combinations may deserve indexable landing pages; most do not.

Calendars and date navigation

“Next month” links can create dates indefinitely into the future or past. Bound the available range and avoid linking to empty periods.

Internal search results

Search forms can generate unlimited URLs from arbitrary text. These pages are rarely suitable as a scalable indexable content strategy.

Tracking and session parameters

Analytics tags, campaign IDs, referral values, and session identifiers can create duplicate URLs for the same content.

Broken pagination

Pages beyond the real result set, negative values, and alternate parameter orders can generate duplicate or empty sequences.

Diagnose before blocking

1. Group URL samples by path and parameter name.
2. Determine whether each variation changes the primary content meaningfully.
3. Identify where the URLs are generated: links, forms, scripts, sitemaps, or external sources.
4. Check response status, canonical target, indexability, and internal links.
5. Decide which combinations should be crawlable and indexable.
6. Fix URL generation and navigation before relying on crawler directives alone.

Blocking a pattern in robots.txt can reduce crawling, but it does not remove already known URLs from search and can prevent crawlers from seeing canonical or noindex signals. Use the directive only after defining the intended URL policy.

Choose a control strategy

• Normalize links: link internally only to preferred URLs.
• Limit combinations: expose useful filters and prevent unbounded combinations.
• Canonicalize duplicates: use a consistent canonical when pages are truly duplicate or near-duplicate.
• Return appropriate status codes: invalid combinations should not produce endless successful pages.
• Remove parameters from sitemaps: list only canonical URLs.
• Use robots rules carefully: reduce crawling of known low-value patterns after understanding index implications.
• Preserve valuable landing pages: do not block all facets when selected combinations satisfy real search demand.

Google's crawl-budget guidance describes faceted navigation, duplicate URLs, soft errors, and unnecessary redirects as common sources of wasted crawling on larger sites.

Example

A category has five filters with dozens of values, and the site links every selected combination to every remaining option. Screpy observes tens of thousands of URLs under one path with different parameter orders. Most pages show the same few products and self-canonicalize.

The fix is architectural: define valuable combinations, normalize parameter order, stop linking invalid combinations, return a correct status for empty states, and canonicalize genuine duplicates. Adding a robots rule without changing the link generator leaves users and other crawlers exposed to the same explosion.

Verify the fix

Sample representative allowed and disallowed combinations. Run a new crawl and compare discovered-URL counts, duplicate patterns, request volume, and crawl completion. Monitor valuable facet pages separately so efficiency improvements do not remove useful organic landing pages.