Screpy's free robots.txt generator creates a plain-text file from your crawler groups, Allow and Disallow rules, and sitemap URLs. Use it to prepare a new file or rebuild a policy you have already planned. Copy the result or download robots.txt, then publish it on your own website.
The generator does not read your existing file, test live URLs or install changes. Before replacing a working policy, keep a copy and identify which paths each crawler should be able to request. A generated file can be syntactically valid while expressing the wrong policy for your website.
How to use the robots.txt generator
Start with the * crawler group for the general policy. Choose Add rule, select Allow or Disallow, and enter a path beginning with /. For example, enter /drafts/ in the path field, not https://example.com/drafts/. Add another group only when a named crawler needs its own policy.
You can configure up to 10 unique crawler groups, 50 explicit rules across all groups and 10 sitemap URLs. Use the crawler's product token, such as Googlebot, rather than a complete browser User-Agent string. Repeated crawler names, even with different capitalization, are not separate groups in this tool.
Put each complete HTTP or HTTPS sitemap address on its own line. Leave this field empty if you do not have an address to supply. Generate the file, review its warnings, then copy or download the output. A group with no explicit rules produces Allow: /; the explicit-rule count can therefore be zero even though the file contains that default line. These are generator limits, not the limits of the robots.txt protocol.
A robots.txt example with a permitted exception
This fictional configuration restricts crawling of a drafts folder while leaving a public subfolder available. It is an example to adapt, not a recommended default for every website.
User-agent: *
Disallow: /drafts/
Allow: /drafts/public/
Sitemap: https://example.com/sitemap.xml
For a crawler using this group, the expected rule decisions are:
| URL path | Crawl decision | Reason |
|---|---|---|
/products/blue-mug/ |
Allowed | Neither restriction matches. |
/drafts/new-page/ |
Disallowed | The blocked folder matches. |
/drafts/public/style-guide/ |
Allowed | The longer Allow path supplies an exception. |
/drafts |
Allowed | It does not contain the final slash in /drafts/. |
The last row matters when your application serves both folder variants. Test the URLs your site actually exposes, including slash and query variations. A rule decision describes permission under the file; it does not establish HTTP availability or search indexing. The sitemap line points to a file location and does not override a Disallow rule.
Match the intended paths with wildcards
Check a rule against both an intended match and a similar URL you want to keep crawlable. Paths are case-sensitive, so /Catalog/ and /catalog/ are different. A missing slash or overly broad prefix can change the affected pages.
| Pattern | What to check before using it |
|---|---|
/catalog |
A prefix: it also matches /catalogue/. |
/catalog/ |
Matches descendants of that folder, but not /catalog by itself. |
/catalog$ |
Matches the exact ending, not /catalog/ or /catalog?sort=price. |
/*.pdf$ |
Matches a URL ending in lowercase .pdf; a query after the extension changes that ending. |
The Robots Exclusion Protocol defines * for a sequence of characters and $ for the end of a match. A literal # starts a comment; encode a hash or space that belongs in a URL rather than entering it as unescaped path text. Keep patterns narrow enough to explain and verify. This form preserves your patterns; it does not calculate every URL they could affect.
Give each crawler a complete rule group
The wildcard group is not a set of restrictions that Googlebot automatically adds to its specific group. For example:
User-agent: *
Disallow: /drafts/
User-agent: Googlebot
Disallow: /previews/
Here Googlebot's group restricts /previews/, but not /drafts/. If both folders should be restricted for Googlebot, add both rules to that group. Google's robots.txt specification explains its group selection and matching behavior. For equally specific conflicting Allow and Disallow rules, Google chooses Allow; moving the same rule further down the file does not make it win.
Add a bot-specific group only when you know the intended distinction. Search, training and user-requested fetchers can use different tokens. Use the AI Bot Access Tester to review a supported bot's rules rather than assuming one group controls every activity associated with a service.
Read the generator's warnings before copying
The report highlights three situations that deserve a deliberate decision. A warning is a prompt to review the policy, not proof that the file is unusable.
| Warning | What to review |
|---|---|
| Whole-site Disallow | A group contains Disallow: /. Confirm the broad restriction is intentional and inspect any more specific Allow exceptions. |
| Specific crawler group | The wildcard group has explicit rules and another crawler has its own group. Check that the latter includes its intended restrictions. |
| Matching Allow and Disallow paths | Both actions use the same path in one group. Remove an unintended conflict; Google prefers Allow for an equally specific match. |
An output without warnings is not a live crawl audit. These checks do not determine whether an asset is needed to render a page, whether a sitemap is reachable or whether every pattern matches the pages you intended. Keep stylesheets, scripts and images crawlable when the page needs them; do not add a blanket asset-folder restriction just because a generic template includes one.
Publish robots.txt and check the live response
Save the current file before replacing it. Publish the generated text as a UTF-8 plain-text resource named exactly robots.txt, without an extra .txt extension. For an HTTPS site at www.example.com, check https://www.example.com/robots.txt. A file under /blog/robots.txt does not configure the whole host. The intended protocol, host and port matter.
On WordPress, another CMS or a hosted website builder, first identify what serves that URL. It might be a physical file, a plugin-generated response or a platform setting. Update the responsible source; placing a second file elsewhere will not necessarily change the live response. Screpy does not provide CMS presets or publish the file for you.
After publishing:
- Open the exact robots.txt address without being signed in and confirm it returns the intended text, rather than a login screen or HTML page.
- Use Robots.txt Tester on important allowed and disallowed URLs. Include exceptions and alternate path spellings.
- Investigate unexpected responses with HTTP Headers Checker and verify a declared sitemap separately with Sitemap Checker.
Google's creation guide explains upload and public-access checks. Crawlers discover the published file; downloading it here is not a submission. A cached copy can also mean a change is not observed immediately.
Choose the right control for indexing or private pages
Use robots.txt to communicate crawl restrictions. If your objective is to keep an otherwise public page out of Google Search, Google documents noindex in a page's meta tag or HTTP response header. Google must be able to access the page to read it. Blocking the same URL in robots.txt can prevent that, and adding a Noindex: line to robots.txt is not supported by Google.
Check existing directives with Noindex Checker before choosing a remedy. For confidential documents or a private staging site, use authentication and appropriate access controls. Neither a public robots.txt file nor a noindex instruction secures the underlying content. Generating more rules is not a substitute for choosing the right control.