For public Nuxt routes, check the robots.txt served from the production host before changing page metadata. A stale imported rule, the wrong environment setting or an unexpected crawler group can block the routes you intended to expose, even when those pages return successful responses.
Nuxt Robots also manages page-level robots tags and response headers, so a permissive robots.txt is only one part of the policy. These seven checks trace the configuration through to the file and page responses actually published.
Module examples target @nuxtjs/robots 7.0.2. Its current installation guide requires a compatible Nuxt 4.6 or Nuxt 5 setup; older Nuxt tutorials can use different options. Check your lockfile before copying configuration. This is a documentation and source-based audit, not a claim that your site was tested or that these checks guarantee indexing.
1. Identify what serves the host-root robots.txt
Start with the exact production origin, such as https://www.example.com/robots.txt. Google's rules apply to that host, protocol and port. A file at /docs/robots.txt does not govern the host's public routes.
Identify the effective owner:
| Setup | What to inspect |
|---|---|
| Root-mounted Nuxt site with a static file | public/robots.txt and the file actually deployed |
| Nuxt Robots module | Enabled module, parsed input file and generated response |
| Host-managed policy | The hosting rule or root site that supplies the public file |
Nuxt documents public assets as files served without build transformations. With the robots module enabled, however, a public robots file can become an input to generation; check 2 covers that distinction.
Inspect the installed dependency tree:
npm ls nuxt @nuxtjs/robots @nuxtjs/seo
Use your package manager's equivalent. The SEO bundle already includes Nuxt Robots, so an example using the standalone module is not a reason to register it twice.
Check subpath deployments carefully. In version 7.0.2's setup code, a non-root app.baseURL disables robots.txt generation. A Nuxt app mounted beneath another site therefore needs a deliberate arrangement for that host's root policy. Do not treat a subdirectory file as a replacement for it.
2. Trace the file and configuration rules being merged
A new configuration can still publish an old restriction. Nuxt Robots can parse an existing file and combine its policy with generated output, so inspect the imported source before adding another override.
The file configuration guide recommends <rootDir>/public/_robots.txt. It also documents other candidates, including a project-root robots.txt and files under the source directory's assets or pages paths. On a default Nuxt 4 layout, those source paths are under app/.
Two details matter:
- With the default import behavior, an existing
public/robots.txtis moved topublic/_robots.txtand merged into the generated response. robots.mergeWithRobotsTxtPathcan select a custom input path; setting it tofalsedisables that file-import behavior.
Treat these as supported input locations, not a promise that every copy is concatenated. Determine which file your installed version selects, including configuration supplied by layers.
Search the selected input and effective configuration for blanket Disallow: / rules, obsolete public-path exclusions, duplicate named-agent groups and stale sitemap addresses. Compare them with the final response.
Preserve intentional restrictions while correcting an unwanted one. Removing a rule from one source is insufficient if another configuration owner still emits it. Equally, disabling file imports without reviewing the imported policy can silently discard rules you meant to retain.
3. Compare production and preview indexing environments
Compare the same public route on production and preview. Read three outputs together: robots.txt, the page's robots meta tag and its X-Robots-Tag response header.
Nuxt Robots uses shared Site Config to decide whether an environment is indexable. The environment guide documents NUXT_SITE_ENV=staging and site.indexable: false for non-public indexing policies.
A global non-indexable setting produces a blanket crawl restriction and noindex directives. Check that this policy has not leaked into the production build.
| Output | Configuration timing to verify |
|---|---|
| Server-generated robots and page responses | Effective server environment and Site Config |
| Prerendered HTML and robots.txt | Values used when those artifacts were built |
| Exported static-host header rules | Build-time configuration, adapter output and the host's actual response |
Changing a runtime variable cannot rewrite already generated HTML or static header rules. The module's implementation guide explains the distinction. A host must also support the exported header format; inspect the public response rather than assuming a generated _headers file was applied.
Keep preview confidentiality separate. Robots directives do not authenticate visitors, and blocking crawling does not reliably remove already known URLs from search results.
For the wider bundle setup and environment audit, use our Nuxt SEO module checks. Here, the pass condition is narrower: production and preview each serve their intended crawl and indexing policy.
4. Separate route indexing rules from crawl blocks
In Nuxt Robots, a robots route rule primarily controls page indexing directives. It is not automatically an equivalent Disallow line.
For example, if your intended policy excludes a catalog area from indexing but keeps its public subtree eligible, a configuration fragment could be:
export default defineNuxtConfig({
routeRules: {
'/catalog/**': { robots: false },
'/catalog/public/**': { robots: true },
},
})
The more specific public rule wins in the module's route resolution. A false rule uses robotsDisabledValue, whose default is noindex, nofollow. The route-rule documentation describes the resulting headers and meta tags.
In version 7.0.2, disallowNonIndexableRoutes is deprecated and defaults to false. Enabling it adds crawl restrictions for non-indexable route rules. Therefore, inspect that option and the generated file when migrating an older configuration.
This distinction matters for a public page that should be removed from search results. Google must fetch it to read a noindex tag or header. A robots.txt block can prevent that fetch; Google's noindex guidance explains why the controls are not interchangeable.
For an intended public catalog page, verify both its selected route rule and its actual response. A public exception does not erase an imported crawler restriction or an unrelated noindex header added by the host. On static deployments, also check how matching header rules are combined.
None of these settings provides access control for a private catalog.
5. Test the selected crawler group against real public paths
Evaluate the file as a crawler would, rather than reading its lines as an ordered firewall list.
Consider this literal robots.txt policy:
User-agent: *
Disallow: /account/
Allow: /account/help/
Before adding other groups or generated locale rules, its expected results include:
| Requested path | Crawl result | Reason |
|---|---|---|
/pricing/ |
Allowed | No matching restriction |
/account/settings |
Disallowed | Matches the account prefix |
/account/help/setup |
Allowed | The longer help rule wins |
/account/helpdesk/ |
Disallowed | Does not match the /account/help/ exception |
/Account/settings |
Allowed by this example | Path case differs |
/fr/account/settings |
Allowed by this example | No matching locale-prefixed rule |
Google chooses the most specific matching user-agent group. A dedicated Googlebot group does not inherit the wildcard group's restrictions. Multiple applicable groups for the same specific agent are combined; the specific group and * group are not combined.
Within the selected group, Google uses the most specific matching path rule and favors the less restrictive rule in a conflict. Its robots parsing specification also explains path case and supported wildcards. Nuxt route globs and robots.txt matching syntax should not be treated as identical.
Repeat the matrix against the final emitted file, including actual case, slash forms, locale prefixes and relevant query URLs. Nuxt Robots can expand rules through its i18n integration, so the unprefixed example above is not a prediction of every localized deployment.
Add real JavaScript, stylesheet and image paths needed by public pages, including relevant /_nuxt/ assets. A broad restriction can interfere with rendering even when the HTML route itself is allowed.
Record the selected group and winning rule for each important case. An ordinary HTTP fetch alone does not perform this policy evaluation.
6. Follow the sitemap declarations to actual XML
Read every Sitemap: line in the published robots.txt. Each should identify an intended, accessible sitemap destination, not a preview domain or an address copied from another deployment.
An illustrative declaration is:
Sitemap: https://www.example.com/sitemap.xml
Use your actual endpoint. A file with that name may be a sitemap index rather than a direct URL list, so inspect its XML root and follow the referenced child sitemaps.
Nuxt Robots supports sitemap configuration and integration with Nuxt Sitemap. If you supply relative sitemap paths in configuration, verify the absolute origin used in the generated file. An explicit absolute address is easier to inspect, but it still needs to match the intended public inventory.
Check at least one approved public route and one deliberately non-indexable route against the resulting sitemap policy. A declaration in robots.txt does not prove that the XML contains the right URLs, or that those destinations are crawlable.
Use our Nuxt sitemap checks for the complete dynamic-source, locale and XML reconciliation workflow. Keep that investigation separate from this check's immediate question: whether robots.txt points crawlers to the sitemap files you actually publish.
7. Capture the deployed file, headers and cache evidence
Fetch the actual public endpoint with a complete anonymous GET, not just a HEAD request:
curl -sS -L --max-redirs 5 -D robots-headers.txt -o robots-response.txt https://www.example.com/robots.txt
Replace the example origin. The command saves response headers, including redirects, and the final body. Inspect the final status, destination, content type and text. An HTML error page with HTTP 200 is not evidence that your intended robots policy was delivered.
Compare the body with the approved policy and repeat the path matrix from check 5. For representative public pages, fetch their HTML and headers too, especially after changing environment or route rules.
Then separate the cache layers. The module configuration reference for Nuxt Robots 7.0.2 documents a default robots response policy of max-age=14400, must-revalidate, or four hours. The deployed host can apply a different policy. Record observed Cache-Control, Age, validators and deployment revision instead of inferring freshness from a local file.
Google generally caches robots.txt for up to 24 hours, with variations described in Google's caching rules. A fresh response in your terminal does not prove that Google has refreshed its cached copy. Likewise, sending a Googlebot user-agent string does not establish that a request came from a verified Google crawler.
Keep a small release ledger. This CSV template uses the saved body filename; observation fields remain blank until checked:
origin,release,final_status,body_file,cache_control,checked_at
https://www.example.com,,,robots-response.txt,,
Our robots.txt guide covers the broader policy distinctions. For this Nuxt release, retain the selected source, final file, header evidence and public-path results together. That establishes what you published and evaluated, while crawler refresh, indexing and rankings remain separate outcomes.