6 Next.js Robots.txt Checks Before Deployment

A Next.js robots.txt check should establish three things: which source produces the file, which crawl policy the release is supposed to serve, and what an anonymous request actually receives. A valid app/robots.ts function can still ship the wrong policy if a production build uses preview settings, an older response stays cached, or a routing layer replaces the text with an HTML page.

The six checks below cover that release path. They focus on the App Router, with notes for an existing Pages Router implementation or a static export. Match the examples to your installed Next.js version and hosting setup; the domains and paths are illustrative.

Robots.txt manages crawling by compliant bots. It does not protect private content or guarantee that a URL stays out of search results. Keep the intended outcome clear before changing a rule.

1. Identify the source that owns /robots.txt

Start with the public endpoint, then trace it to its source. On an ordinary HTTPS origin, the relevant address is https://example.com/robots.txt, with the plural filename. A file available only at /en/robots.txt, /public/robots.txt or an application subdirectory is not the origin's root robots file. If an app lives beneath a base path, explicitly verify how the host serves the root address rather than assuming the app's mount point solves it.

Use this ownership map to inspect an existing project:

The current App Router robots convention supports a static text file or a default-exported function returning MetadataRoute.Robots. For a Pages Router project, the public folder documentation describes serving static files from the site's root.

For example, this is a minimal generated production policy, not a universal block list:

// app/robots.ts
import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: {
      userAgent: '*',
      allow: '/',
      disallow: '/account/',
    },
    sitemap: 'https://example.com/sitemap.xml',
  }
}

Replace the fictional domain and choose paths from your own crawl policy. This example assumes the sitemap exists; the field does not generate it. Account data still requires authentication.

Before adding this file, search for existing robots sources, post-build generators and host-level overrides. Two implementations aimed at the same address are a conflict to resolve deliberately, not a useful fallback arrangement. Do not assume an undocumented winner or remove an older source without checking which deployments use it.

Record one owner for the endpoint and the release revision containing it. The later checks evaluate that owner's output.

2. Separate production policy from preview protection

Write down the expected behavior for each deployment before inspecting an environment variable. The same source can produce different files during separate builds, and a candidate deployment may run optimized production code without being the public website.

Do not use NODE_ENV === 'production' as proof that a deployment should be indexed. The Next.js environment variable guide explains that, when unset, Next.js assigns development for next dev and production for other commands. A preview can therefore be built in production mode too.

Inspect the actual deployment classification, publication hostname and values used when generating the file. If a robots policy was evaluated into a build artifact, changing a runtime label does not rewrite those already-generated bytes. Likewise, a browser-exposed NEXT_PUBLIC_ value is frozen at build time; its name does not make it follow a later domain promotion.

For Vercel projects, the current environment documentation describes staged production deployments that can be verified before promotion. Promotion assigns production domains without rebuilding. That makes the artifact's original inputs relevant even after the public address changes. Other hosts need the equivalent check against their own deployment process.

Keep preview exclusion and private access separate. For a publicly retrievable page, Google's noindex guidance requires the crawler to be able to fetch the directive. Blocking that page in robots.txt can prevent Google from seeing it. Sensitive preview content needs access controls instead.

Check the release candidate before switching traffic, then confirm the public domain after activation. A preview hostname proves only its own response; it cannot establish what a different origin will serve.

3. Evaluate crawler groups against a path matrix

A rule can be syntactically valid and still allow the wrong URL. Review the complete file for the crawler you mean to control, then compare representative paths with written expectations.

Google selects the most specific applicable user-agent group. It does not add the wildcard group's rules to a specific Googlebot group. This is a consequential distinction in Google's robots specification: adding a Googlebot section can change behavior even when the original User-agent: * section remains untouched.

Consider this deliberately diagnostic fixture, not a recommended production policy:

User-agent: *
Disallow: /account/
Disallow: /preview/

User-agent: Googlebot
Disallow: /account/
Allow: /account/help/

Expected permissions for the fixture are:

The last two rows expose a policy mismatch if /preview/ should be disallowed for both groups. Add the restriction to the specific group as well, or remove the unnecessary separate group after reviewing its purpose. Neither choice supplies authentication or reliable index removal.

Path matching is a prefix comparison, not a filesystem directory lookup. RFC 9309 specifies using the most specific matching path and recommends case-sensitive matching and favoring allow for equivalent rules. The fixture follows those rules, consistent with Google's documented interpretation. For example, changing /account/ to /account also matches /accounting. Check both the intended match and a nearby URL that must remain accessible.

Build your own matrix from actual routes: a public article, an excluded path, a slash boundary, a required asset and any explicit exception. Include each relevant crawler separately. A rule evaluator can check permissions for those inputs; it cannot prove the destination returns useful content, that a real crawler fetched it, or that it is indexed.

Keep the fixture and actual results with the release. A generic verdict that the file is valid does not replace these URL-level expectations.

4. Keep required resources and noindex directives reachable

Do not copy a blanket Disallow: /_next/ rule into every Next.js deployment. That path can contain JavaScript, styles and other resources used by public pages. Blocking a resource is a problem when Google needs it to understand the page, as Google's robots.txt introduction explains.

Inspect dependencies of representative public pages before excluding a resource path. The same caution applies to /api/: a route that only processes an authenticated mutation has a different role from a publicly readable endpoint needed to load meaningful page content. Directory names alone do not establish the correct crawl policy.

Choose the mechanism that matches the goal:

For a page intended to be excluded with noindex, verify both its directive and its crawl permission. Do not add a broad disallow and assume the combination is stronger. For a page intended to be indexed, inspect page-level meta tags and X-Robots-Tag headers too; allowing crawling does not cancel an independent indexing restriction.

Google's JavaScript SEO guidance also warns that an initial noindex can cause Google to skip rendering. Removing it later with client-side JavaScript is not a dependable release fix.

The release evidence should therefore distinguish three observations: the URL is permitted by the rules, its required response can be retrieved, and the page's indexing directives match its intended state. If one fails, record that specific failure. Use Screpy's general robots.txt guide for broader crawl-policy decisions; this check is about preserving the access your Next.js release actually needs.

5. Verify the sitemap destinations in the output

Read every Sitemap: line in the generated text. Each should point to the intended, fully qualified sitemap or sitemap-index address. Check the actual file at that address, not just the value in robots.ts.

For a fictional site, this declaration is meaningful only if the referenced file exists and contains the intended discovery data:

Sitemap: https://example.com/sitemap.xml

Look for a preview hostname, localhost, the wrong protocol, a discarded subdomain or a path that the current release no longer serves. A hardcoded production origin can be appropriate for a single-origin policy; a multi-domain setup needs an explicit mapping to the correct destination rather than whatever host happened to run the build.

Next.js accepts a sitemap address or an array of addresses in the robots metadata object. That capability does not establish that /sitemap.xml is automatically available, that every child sitemap is listed, or that its URLs agree with your crawl policy. For generation, shard discovery and root-index checks, use the separate Next.js sitemap mistakes guide.

A sitemap declaration does not override Disallow or a page's noindex. If a URL is meant to be public and indexable but the release lists it while blocking it, resolve that disagreement deliberately. If exclusion is intentional, keep the discovery inventory consistent with that decision.

Keep the sitemap address, response status and observation time in the same record as the robots response. Finding a valid declaration is a discovery check, not evidence that Google has fetched or indexed the listed pages.

6. Verify the built and deployed response after caching

A generated function and a fresh HTTP response are different pieces of evidence. The current Next.js robots convention is cached by default unless it uses a request-time API or a dynamic configuration option. An async function or an environment lookup alone is not evidence that the file regenerates on every request.

For a static export using output: 'export', inspect the completed export and the files your host will serve. The static export guide describes build-time output and the limits on request-dependent behavior. An exported file cannot discover a new runtime environment on a static server. A custom GET handler intended for export must satisfy the installed version's prerendering requirements; the current guide explicitly shows dynamic = 'force-static' for those handlers. Do not apply that setting blindly to a server-hosted implementation with different requirements.

Next.js runtime redirects, headers and Proxy are not supplied by a pure static export. Equivalent delivery behavior, when needed, belongs to the actual host. Verify the exported robots file is present at the path that host publishes, and that a generic HTML fallback does not replace it.

For a server deployment, inspect routing layers as well. In current Next.js, Proxy runs before filesystem routes; older projects may call it Middleware. A login redirect, locale rewrite or catch-all can intercept /robots.txt even when the source file exists. Narrow any correction to the intended public metadata route. Do not disable access controls for private application routes.

Capture a complete anonymous GET

These read-only commands use a fictional origin and save the first response without following redirects. Run them in a shell with curl, against an environment you are authorized to inspect:

curl --silent --show-error --max-time 30 \
  --dump-header robots-headers.txt \
  --output robots-body.txt \
  'https://example.com/robots.txt'

curl --silent --show-error --max-time 30 \
  --dump-header page-headers.txt \
  --output page-body.html \
  'https://example.com/blog/release-notes'

Read the status, headers and complete body. A successful delivery check means the expected UTF-8 plain-text rules are returned, normally with Content-Type: text/plain, rather than a login page or an HTML error with status 200. If redirected, inspect Location, confirm the intended destination and fetch it separately. Do not interpret the first response as the final policy.

An unavailable robots endpoint is not a safe replacement for a deliberate rule. Google treats most robots.txt 4xx responses other than 429 as no crawl restrictions; server errors follow different handling. Do not use an error response to simulate Disallow.

Keep freshness and crawler evidence separate

Compare the observed file with the intended release output and repeat the path matrix from check 3 against that observed text. If an older policy appears, identify whether it came from the build artifact, application cache or delivery cache before choosing a correction. Preserve available cache headers and the deployment revision; avoid guessing that changing one TTL fixes every layer.

Google also caches robots.txt independently. A corrected anonymous response does not prove Google's cached copy has refreshed. Search Console's available crawl and live-inspection evidence can help investigate that distinction, but a local request with a Googlebot header is not proof of a genuine Googlebot fetch.

Use this CSV header as a reusable release record:

observed_at_utc,release_revision,environment,requested_origin,robots_source,robots_status,robots_content_type,robots_artifact_reference,crawler_token,tested_path,expected_permission,observed_permission,page_status,page_robots_directive,follow_up_owner

Keep the saved response files beside that record. Mark an observation unknown when it was not available; do not convert an unchecked URL into a pass. Recheck the candidate before activation and the real production origin afterward, especially when domains are assigned to an existing artifact. For conflicts in page-level tags, continue with the Next.js metadata guide.

This is an AI-assisted, documentation-based workflow checked against current first-party references on October 29, 2026. The examples are illustrative and were not executed against a customer application. Passing these checks establishes a documented crawl-policy release result; indexing and rankings remain separate outcomes.