Use the Bulk Meta Robots Checker to compare noindex and nofollow directives across up to 20 public URLs. It reads HTML robots tags and X-Robots-Tag response headers, then groups the observed results for your selected crawler scope. This helps you spot an unexpected exclusion across several page templates or verify deliberate exclusions after a change. The report describes the responses the checker could read; it does not establish whether Google has indexed those pages.
Choose a useful batch and crawler scope
Start with the pages affected by the change you want to check. After editing a product template, include several product URLs and a category page. After a migration, include important landing pages and any URLs meant to redirect. A small representative batch helps you compare behavior between page types; it cannot establish that the entire site behaves the same way.
Decide which pages should be eligible for search and which have an intentional exclusion. For example, a public product page and a thank-you page may have different requirements. A noindex warning needs attention when it conflicts with that requirement, rather than simply because it carries a warning icon.
Paste complete HTTP or HTTPS URLs, one per line, with no more than 20 non-empty lines. Repeated normalized URLs are checked once, but remove duplicates before submitting a list that exceeds the limit. The tool checks the supplied addresses; it does not discover more pages from your domain or import a sitemap.
Select Googlebot to interpret general declarations together with Googlebot-specific declarations. Generic focuses on general declarations. This selection changes which instructions apply in the report. It does not turn the request into a real Googlebot visit or prove that Google receives an identical response.
Read noindex, no noindex observed and unknown results
The summary and filters group URLs by the observed noindex outcome. The Noindex warning means an applicable noindex instruction was found. The green No noindex observed badge means no such instruction was found with complete applicable source coverage. Unknown means the checker could not establish a negative result. Green does not confirm indexing, and a neutral unknown badge does not mean the page permits it.
Read the HTTP status, final URL and evidence beside that outcome. If a submitted URL redirects, the directives describe the final response reached by this check. A status code alone does not tell you whether a robots declaration is present. Also read the separate nofollow value; the summary is not a combined score for both directives.
These illustrative cases show how page intent changes the next action. They are not results from a scan of a real website.
| Page intention | Evidence read | How to act |
|---|---|---|
| Public product should appear in search | Applicable noindex found |
Locate its tag or header and investigate the unintended exclusion |
| Thank-you page should stay out of search | Intended noindex found |
Confirm that the declaration and crawler scope match the requirement |
| Public article should appear in search | No noindex; sources fully read |
Continue other indexing checks if the article is missing from search |
| Any page, source unreadable | No confirmed noindex; coverage incomplete |
Investigate the missing evidence before changing instructions |
A partial response can still provide a definite positive: a readable header containing noindex remains evidence even if the HTML cannot be read. The opposite conclusion needs adequate coverage. For a PDF or another non-HTML resource, HTML coverage can be Not applicable; the relevant source is its response header. Not applicable and unavailable describe different situations.
Check HTML tags and X-Robots-Tag together
An HTML tag and a response header can carry different instructions. For example, suppose the final HTML contains:
<meta name="robots" content="index, follow">
Its response also contains:
X-Robots-Tag: noindex
Adding index to the HTML does not neutralize the applicable header's noindex. Google's robots directive reference explains how multiple declarations combine and restrictive instructions resolve conflicts. In this example, inspect the header setting instead of repeatedly changing the HTML tag. The report preserves each declaration's source, crawler and raw value so you can find the relevant instruction.
Scope matters too. A general robots tag saying index, follow can coexist with a googlebot tag saying noindex. The Googlebot selection includes the specific exclusion; the generic selection considers the general declarations. Different results are therefore possible without either run being a measurement of actual Google indexing.
The shorthand none means both noindex and nofollow. Other declarations may need a more detailed interpretation: for example, indexifembedded concerns content embedded elsewhere. This batch summary records the two flags; it is not a complete decision about every directive, embedding situation or search feature. Keep the raw evidence when an exception needs investigation.
Keep nofollow separate from indexing
Noindex concerns the page's inclusion in search results. Page-level nofollow concerns following links on that page. A declaration of noindex, follow is not inherently contradictory: the owner may want the page excluded while leaving its links available for discovery. Review these choices against the page's purpose rather than treating every limiting directive as an error.
The batch checker reports page-level declarations. It does not examine each link's rel="nofollow", sponsored or ugc attribute, so a No result for page-level nofollow does not establish how every individual link is marked.
Google's link qualification guidance also explains that linked URLs can be discovered through other sources. A nofollow observation here does not prove those destinations will never be crawled or indexed. If your question is whether this page itself has an exclusion, return to its noindex evidence.
Investigate unknown results before changing directives
Begin with the coverage and evidence, not the assumption that a tag is missing. A failed request, unreadable document or unsupported declaration can leave the checker unable to decide. If there is a final URL, check that it is the destination you intended. A redirect to a login page or an access-denied response is not evidence about the original page's normal public content.
This checker reads the HTTP response; it does not run JavaScript to build a rendered page. Instructions inserted only after scripts execute are outside that view. Likewise, a page that requires a login, blocks automated access or varies its response by visitor may not expose the content you see in your browser. Do not remove useful directives merely to turn an unknown row green. First establish what could actually be read, then retry when the public response is available.
Use the Noindex Checker when one URL needs a closer look at declarations and exceptions. Keep the batch result as the comparison record while investigating that page separately.
Crawl permission is another question. This tool does not read robots.txt; use the Robots.txt Tester for those path rules. Google needs access to a page to observe its noindex instruction. Combining a crawl block with an exclusion therefore needs a separate access check, even when this checker can fetch the URL.
Export findings and recheck the same pages
Filter to the group you need to investigate before downloading CSV. The CSV contains the visible rows' URL and status classification. It is a compact work list, not a complete dump of robots evidence: it does not include each raw declaration, the nofollow value or all response details. Download the full report as JSON when the recipient needs the detailed evidence and coverage information.
For a reproducible correction:
- Keep the submitted URLs, selected crawler scope and original report together. Record each page's intended behavior alongside the findings.
- Identify the declaration's source. Correct an unintended HTML tag in the relevant page or template; correct an unintended header in the setting that publishes that header. Retain exclusions that are deliberate.
- Run the same batch and scope again after the updated public responses are available. Compare the raw evidence as well as the counts; an unchanged total can conceal a different set of URLs.
- Investigate any remaining unknown rows separately rather than counting them as resolved.
If the revised response is correct but Google's result has not changed, this snapshot alone cannot explain its processing state. Google's noindex troubleshooting guidance points to URL Inspection and Page Indexing reports for checking what Google received. A successful recheck confirms the current observed response, not an immediate recrawl or a promised search result.