An llms.txt file is a proposed Markdown document that points language-model tools to a curated set of useful pages on a website. It can give an AI system a concise overview and a cleaner route to documentation, policies, products, or other canonical resources.
The important limitation is simple: llms.txt is a proposal, not a universal web standard or a guaranteed SEO signal. Publishing one does not force an AI crawler to fetch it, does not grant permission to crawl blocked content, and does not replace normal technical SEO.
What is llms.txt?
The llms.txt proposal describes a UTF-8 Markdown file placed at the root of a site:
https://example.com/llms.txt
The file normally contains:
- One H1 with the site or project name.
- A short blockquote summary.
- Optional explanatory text.
- H2 sections containing lists of important links and concise descriptions.
- An optional section for lower-priority resources.
Unlike a sitemap, the goal is not to enumerate every indexable URL. The value comes from curation: selecting the pages that best explain the entity, product, documentation, or subject area.
Is llms.txt an official standard?
No. It is an open proposal maintained in the llms.txt specification repository. Different tools may support it, ignore it, or interpret it differently. There is no guarantee that a search engine or AI product will use the file simply because it exists.
That status changes how you should prioritize the work:
- Fix crawlability, canonicalization, internal links, rendering, and content quality first.
- Treat llms.txt as a low-cost discovery aid for tools that choose to support it.
- Measure actual crawler behavior instead of assuming adoption.
If a vendor documents a specific crawler or control mechanism, follow that documentation. For example, OpenAI publishes separate information about its web crawlers and user agents. Those controls are distinct from the llms.txt proposal.
llms.txt vs. robots.txt vs. sitemap.xml
| File | Primary purpose | Standardized behavior | Should contain |
|---|---|---|---|
robots.txt |
Crawling rules for compliant user agents | Yes, through the Robots Exclusion Protocol | Allow and disallow rules, plus supported directives |
sitemap.xml |
Discovery of canonical URLs and optional metadata | Yes, through the sitemap protocol | Canonical, indexable URLs you want search engines to discover |
llms.txt |
A curated, human-readable map for language-model tools | No universal adoption guarantee | A short overview and links to the most useful canonical resources |
Do not use llms.txt to hide private material. Do not place secrets, unpublished URLs, customer data, API keys, or internal instructions in it. The file is public.
Does llms.txt improve SEO or AI visibility?
There is no basis for treating the presence of llms.txt as a direct ranking factor. It also cannot compensate for weak pages, unclear authorship, duplicated content, blocked crawling, or poor internal linking.
Its plausible benefit is narrower: a supporting tool may use the file to discover a preferred page or obtain a concise map of a large documentation set. Whether that affects citations or answers depends on the tool, its retrieval process, and the quality and accessibility of the linked pages.
Use outcome-based language when reporting results. “The file was published” is an implementation event. Evidence of impact would be a supported crawler requesting it, discovering linked URLs, or a measurable change in qualified referrals or citations after other causes are considered.
When is an llms.txt file useful?
The proposal is most practical when a site has a clear set of authoritative resources that are otherwise spread across a large information architecture. Examples include:
- Developer documentation and API references.
- Product capabilities and integration guides.
- Research, policy, legal, or institutional resources.
- Support knowledge bases with stable canonical answers.
- Publications with well-maintained topic hubs.
A small brochure site with five obvious pages may gain little. A file that repeats the navigation without adding descriptions or prioritization also offers limited value.
How to create llms.txt
1. Define the audience and scope
Decide what the file is helping a system understand. A company-wide file might cover the organization, products, documentation, policies, and support. A documentation-specific file should focus on guides and references rather than every marketing URL.
2. Select canonical sources
Choose pages that are current, indexable, self-contained, and internally consistent. Prefer a canonical guide over several near-duplicate posts. Exclude tag archives, filtered URLs, tracking parameters, temporary campaigns, thin pages, and content that requires a session.
Before publishing the file, run a website audit and resolve broken destinations. A curated map loses trust quickly when its links redirect repeatedly or return errors.
3. Write concise descriptions
Each link description should explain what the destination provides, not repeat its title. Mention the intended task, coverage, or output in one sentence.
4. Use a simple Markdown structure
# Example Analytics
> Privacy-first analytics documentation for product teams and developers.
Use these resources to understand the product, implement tracking, and troubleshoot integrations.
## Start here
- [Product overview](https://example.com/product): Capabilities, supported use cases, and current limitations.
- [Quickstart](https://example.com/docs/quickstart): Install the SDK and send the first event.
## Reference
- [API reference](https://example.com/docs/api): Endpoints, authentication, request fields, and response formats.
- [Privacy policy](https://example.com/privacy): Data handling, retention, and user rights.
## Optional
- [Changelog](https://example.com/changelog): Product and API changes by release date.
Keep examples inside code blocks so placeholder URLs are not mistaken for real links by users or crawlers.
5. Publish and validate
Serve the file from /llms.txt with a successful response and a text content type. Confirm that it is publicly reachable without authentication, every listed URL is canonical, and Markdown syntax is readable as plain text.
Do not redirect the file through several URLs. If the site has more than one hostname, publish on the canonical host and keep hostname behavior consistent.
What should you include?
Use a small set of sections that reflect real user journeys. For a SaaS product, that might be:
- Product overview and core feature pages.
- Getting-started documentation.
- API and integration references.
- Security, privacy, and data-processing policies.
- Pricing or plan limits when they affect use.
- Troubleshooting and status resources.
Every linked page should clearly identify its subject, provide a useful answer in the rendered HTML, and link to the next relevant step. The file should complement—not substitute for—a coherent internal linking structure.
What should you leave out?
Avoid:
- Every URL in the sitemap.
- Duplicate HTTP, HTTPS, www, non-www, or parameter variants.
- Search results and faceted-navigation combinations.
- Staging, preview, account, and private URLs.
- Pages that exist only to target a keyword without solving a distinct need.
- Claims about AI support that the relevant provider has not documented.
More is not automatically better. A shorter file with strong canonical choices is easier to maintain and interpret.
Maintenance and change control
Assign ownership. Review the file when navigation, canonical URLs, documentation versions, product names, policies, or key workflows change. A quarterly review may suit a stable site; a frequently changing documentation platform should validate it during releases.
Automated checks can confirm that:
- The file returns a successful response.
- Listed URLs return successful responses without redirect chains.
- URLs use the canonical hostname and format.
- No blocked, noindex, private, or duplicate pages are included.
- Required sections and descriptions are present.
Automation should detect drift, but a human should decide which resources deserve priority.
How to measure whether it helps
Start with server or edge logs. Track requests to /llms.txt, the requesting user agent, response status, and subsequent requests to linked pages. Screpy’s AI bot traffic monitoring can help distinguish supported AI crawler activity from ordinary visits.
Use a before-and-after annotation and avoid attributing every visibility change to the file. Content updates, new links, brand demand, search-engine changes, and crawler policy changes can happen at the same time.
Useful evidence includes:
- Verified requests to the file from a documented crawler.
- Discovery of newly linked canonical resources.
- Fewer requests to obsolete or redirected documentation URLs.
- Qualified referral traffic or cited answers that lead to the selected pages.
An absence of requests is also useful: it shows that the file should not receive priority over proven technical and content work.
Security and privacy checklist
Because llms.txt is public, review it like any other published page:
- Include only public canonical URLs.
- Do not reveal unlisted admin, staging, or internal API paths.
- Do not include tokens, credentials, customer identifiers, or confidential instructions.
- Keep access controls on the underlying resources; the file is not an authorization layer.
- Use robots.txt and vendor-documented controls for crawling policy rather than inventing llms.txt directives.
Common questions
Does llms.txt replace robots.txt?
No. robots.txt communicates crawl rules to compliant user agents. llms.txt is a proposed content map and does not grant access to disallowed resources.
Does it replace a sitemap?
No. A sitemap is designed for broad URL discovery. llms.txt should remain curated and explanatory.
Should every website publish one?
Not necessarily. Publish it when you have a meaningful set of authoritative resources and can maintain the file. Do not delay higher-impact crawl, indexation, content, or internal-link fixes to create it.
Should the file contain full page content?
Usually no. The root file should stay concise and link to canonical resources. Some implementations also expose Markdown representations of individual pages, but support and naming conventions vary by tool.
Can it guarantee inclusion in an AI answer?
No. Retrieval, training, citation, and answer-generation policies belong to each system. The file can provide a clearer route to content, not compel its use.
Bottom line
llms.txt is a lightweight experiment worth considering after the fundamentals are sound. Build it as a concise, maintained map of authoritative content; do not market it as a ranking shortcut or a universal AI-control standard. Publish, observe actual crawler behavior, and keep investing in accessible pages, clear authorship, useful content, and strong internal links—the signals every visitor can use today.