What Is llms.txt and Why Your Site Needs One
Contents
llms.txt is a plain-Markdown file hosted at the root of a website (yourdomain.com/llms.txt) that gives AI crawlers and language models a curated map of a site's most important content. Proposed in September 2024 by Jeremy Howard, it is not an official web standard, and no major AI provider has confirmed using it to influence rankings or citations.
This piece breaks the topic into three parts: what the file actually is and where it came from, who is verifiably using it today, and whether creating one is worth your time. Every claim below traces back to a primary source, Google's own documentation, Chrome's engineering team, or a company whose llms.txt file is live and checkable right now, rather than to a secondhand summary. For the wider context of how AI systems decide what to cite at all, the field of generative engine optimization covers the strategy llms.txt fits inside.
What Exactly Is an llms.txt File?
An llms.txt file is a Markdown document with a fixed structure: an H1 with the site or project name, a blockquote summary, optional context paragraphs, and H2-delimited lists of links to the site's most useful pages. It lives at a single, predictable address, yourdomain.com/llms.txt, so any crawler or agent that already knows the convention can fetch it without being told where to look.
The format was proposed by Jeremy Howard, co-founder of Answer.AI, in a post published on September 3, 2024, and the reference specification still lives at llmstxt.org. That specification is the primary source every other article on this topic ends up citing, directly or indirectly, because it is the only document that actually defines the syntax: what counts as a valid H1, what the blockquote is for, and how link sections should be organized.
Nothing about the format is proprietary or gated behind a company. Any site, on any CMS or custom stack, can publish the file as a static text response at the root path. That openness is part of why the convention spread faster among developer tools than among traditional marketing sites, a pattern covered in more detail in the adoption section below.
In practice, the H1 line is almost always just the name of the site or product, nothing more. The blockquote that follows is one to three sentences, written as a plain description an outside reader would understand without prior context, not marketing copy. Everything below that point is organized into H2 sections, each one a logical category (Docs, Product, Company, Blog), with individual links formatted as standard Markdown link syntax followed by a short, factual note about what that specific page contains.
How Is llms.txt Different From robots.txt and sitemap.xml?
robots.txt controls what crawlers may access, sitemap.xml lists every indexable page for search engines, and llms.txt curates a short list of a site's best pages specifically for AI systems to read at inference time. The three files answer three different questions: what's allowed, what exists, and what matters most.
| File | Purpose | Audience | Access control? |
|---|---|---|---|
| robots.txt | Tells crawlers what they may or may not fetch | Search engine bots (Googlebot, Bingbot) and AI crawlers (GPTBot, ClaudeBot) | Yes, via Allow/Disallow directives |
| sitemap.xml | Lists every URL a site wants indexed | Search engine bots | No, it's a discovery list, not a permission system |
| llms.txt | Curates the pages an AI system should read first | Language models, AI agents, RAG tools | No, it contains no Allow/Disallow syntax at all |
A common mistake is treating llms.txt as a permissions file, adding Allow or Disallow style directives to it. It has no such syntax. If the goal is to block an AI crawler entirely, robots.txt is the only file that does that job, using directives like User-agent: GPTBot or User-agent: Google-Extended. Blocking a bot in robots.txt and listing a page in llms.txt are not in conflict, one grants or denies access, the other explains what's worth reading once access is granted. For a broader look at how structured signals help machines parse a page beyond just this one file, schema markup built for AI covers the complementary side of the same problem.
For most site owners, the practical takeaway is simple: if a page shouldn't be seen by any automated system at all, that's a robots.txt decision. If a page should be indexed and findable through normal search, that's a sitemap.xml decision. If a page is one of the handful worth surfacing to an AI system trying to understand the site quickly, that's an llms.txt decision. Treating any one of the three as a substitute for the others is where most implementation mistakes start.
The pattern also isn't unique to text files. Search engines and AI systems alike have always needed some kind of signal about what to crawl, what to index, and what to prioritize. llms.txt is simply the newest entry in that lineage, aimed specifically at systems that read a page's content rather than just its links, which is why treating it as a copy of robots.txt tends to create confusion rather than clarity.
What Does an llms.txt File Look Like?
A minimal, valid llms.txt file is short: one H1 line with the project name, one blockquote with a one-sentence description, and one or two H2 sections that each group a handful of links with a short note on what each page covers. Here's a generic, reproducible example.
# Acme Analytics
> Acme Analytics is a product analytics platform for B2B SaaS teams.
> This file lists the pages an AI assistant should read first to
> understand what Acme does and how to use it.
## Docs
- [Getting Started](https://acme.example/docs/getting-started): Setup and first dashboard
- [API Reference](https://acme.example/docs/api): Full REST API endpoints and auth
## Product
- [Pricing](https://acme.example/pricing): Plans and usage-based limits
- [Changelog](https://acme.example/changelog): Recent product updates
The specification also defines a second, longer variant, /llms-full.txt, which compiles a site's full documentation into one Markdown document instead of a curated list of links. Most sites only need the short version, the full variant matters mainly for large documentation sets where an AI tool benefits from ingesting everything in a single request rather than following dozens of separate links. Both variants sit at the same root level and use plain Markdown, no JSON, no XML, no custom syntax. For a wider set of real examples across different types of sites, the collection of llms.txt examples shows how documentation platforms, SaaS products, and content sites each structure the file differently.
Reading the example above line by line: the H1 is the product name, nothing else. The blockquote gives two sentences of plain context, who the product is for and what the file is meant to do. The "Docs" section groups the two links a developer needs first, setup and API reference, each with a short note instead of a bare URL. The "Product" section groups commercial pages, pricing and a changelog, that matter to a different kind of visitor. Nothing here is complicated; the format rewards restraint over completeness.
A few formatting slips are worth flagging: forgetting the leading "# " on the H1 line, skipping the blockquote and jumping straight to link lists, or nesting H3 headings inside the link sections where the specification expects a flat H2 structure. None of these breaks a site or triggers a penalty, since nothing enforces the format, but a malformed file is simply less useful to whatever is trying to parse it.
Does llms.txt Actually Improve AI Visibility or Google Rankings?
As of its documentation update on June 29, 2026, Google states directly that you don't need to create new machine-readable files, AI text files, markup, or Markdown to appear in Google Search, and that Google Search ignores files like llms.txt entirely, for better or worse. That's an explicit, dated, on-the-record position, not an inference from silence.
Google's John Mueller has made the same point in plainer terms. Asked directly about llms.txt, he said: "AFAIK none of the AI services have said they're using LLMs.TXT [...] To me, it's comparable to the keywords meta tag," a comparison to a field search engines stopped using for ranking decades ago (reported by Search Engine Journal). The comparison is deliberate: a field anyone can fill in, unverified, and with no confirmed downstream effect.
There's one real nuance to the "Google ignores it" story. Chrome's Lighthouse tool shipped an "Agentic Browsing" audit category in version 13.3.0 on May 7, 2026, and that category includes a check for the presence of an llms.txt file, an optional audit that returns N/A if the file is missing rather than a failing score (source: developer.chrome.com). That doesn't mean Google Search itself uses the file for ranking, it means the Chrome team is watching the same standard from the angle of agentic browsing, separate from classic search indexing. For the broader mechanics of how AI systems actually decide what to surface and cite, AI search optimization covers the levers with confirmed effect.
Put together, these three data points paint a consistent picture rather than a contradiction. Google Search itself does not read the file for ranking purposes, full stop. Chrome's engineering team is nonetheless building tooling around the same convention for a different purpose, agentic browsing rather than classic indexing. Anyone claiming a confirmed SEO ranking boost from adding llms.txt is going beyond what any of these three primary sources actually say.
This article, Blue could have written it for you: content optimized for Google + AI, without you writing a single word.
Try mentionLABWho Is Actually Using llms.txt Today?
Adoption remains concentrated among developer-facing companies. Mintlify auto-generates the file for every hosted documentation site it serves, a rollout that started in November 2024, and Anthropic publishes its own llms.txt for its developer platform. That's the clearest, most checkable adoption pattern available right now, not a marketing claim.
Since that November 2024 rollout, documented adopters generating their /llms.txt and /llms-full.txt files through Mintlify's platform include Anthropic, Windsurf, and Bolt.new (source: mintlify.com/blog). What these three have in common is a documentation-heavy, API-first product, exactly the kind of site the format was designed for: dense reference material that benefits from a curated entry point rather than dozens of individual pages.
The reason documentation platforms moved first shows up in traffic data. On sites hosted by GitBook, the share of pageviews coming from AI systems rather than human visitors rose from roughly 9% in January 2025 to 41 to 42% in December 2025, an eleven-month jump that reframes documentation as something increasingly read by machines first. Anthropic's own file backs this up in practice: its developer platform serves an active llms.txt, and a request to the old docs.anthropic.com/llms.txt address returns a live 301 redirect to platform.claude.com/docs/llms.txt, confirmed by checking the URL directly rather than taking a screenshot on faith.
None of this means adoption is limited to those three named companies, it means those are the adopters that are independently checkable right now, through a live redirect, a vendor's own blog post, or a public platform feature. That's a deliberately higher bar than repeating a "brands are switching to llms.txt" claim without a way to confirm it, and it's the bar this article holds itself to throughout.
Should You Create an llms.txt File for Your Site?
The decision comes down to three questions: does your site carry substantial technical or reference content, is there a real audience of AI agents or assistants that would benefit from a curated map, and can you commit to keeping the file current as pages change. If you run heavy documentation, an API reference, or a large knowledge base, the file is a low-cost addition. If your site is a handful of marketing pages, the upside is close to zero, since Google Search ignores it and no other AI provider has confirmed a ranking or citation benefit.
An llms.txt file is not, on its own, a visibility strategy. It's a small technical convenience that sits underneath a much larger set of decisions: how content is structured, how claims are sourced, how topics are covered in depth rather than in scattered fragments. The wider framework for earning citations and visibility across AI systems, generative engine optimization, is where the actual leverage lives; llms.txt is a quick, low-effort task inside that framework, not a substitute for it.
In concrete terms: a SaaS company with an API reference, a developer portal, and a knowledge base is a good candidate. A local service business, a single-location restaurant site, or a five-page portfolio site is not, there's simply not enough distinct, high-value technical content to curate into a meaningful list. Somewhere in between sit content-heavy blogs and media sites, where the file can act as a curated table of contents for AI systems, even without an API to document.
The cost side of this decision is low regardless of which way it goes. Writing a first version of an llms.txt file for a mid-size site takes under an hour once the list of key pages is chosen, and hosting it is a one-time technical task on most stacks. That asymmetry, low effort against an unproven but plausible upside, is a large part of why developer tools adopted it quickly while marketing sites have been slower to bother.
How Do You Create and Publish an llms.txt File?
Creating an llms.txt file comes down to three steps: identify the pages that best represent your site, write them into the official Markdown structure, and host the result at the root of your domain so it's reachable at yourdomain.com/llms.txt. Each step has a few details worth getting right, which is why a full walkthrough is more useful than a summary alone.
The first step, picking pages, matters more than the writing itself: a short list of genuinely useful pages, docs, API reference, pricing, core product pages, beats a long list padded with low-value URLs. The second step means following the exact structure the specification defines, H1 for the name, blockquote for the summary, H2 sections for grouped links. The third step is a hosting detail that varies by platform, static file, CMS route, or server redirect, and is easy to get wrong on the first try. The full step-by-step tutorial for creating an llms.txt file walks through each of these with platform-specific instructions.
One detail that's easy to skip: llms.txt is not a set-and-forget file. When a site adds a new major doc section or retires an old product page, the file should be updated to match, otherwise it starts pointing an AI system toward outdated or missing content, which defeats the purpose of curating it in the first place. There's no fixed update schedule required by the specification, but tying it to the same release process used for documentation keeps it accurate without adding a separate task to remember.
After publishing, the fastest sanity check is opening yourdomain.com/llms.txt directly in a private browser window. The response should be plain text or Markdown, not a 404 page and not a forced download. If a CMS or CDN is serving it with the wrong content type, an AI crawler may fail to parse it correctly even though the file technically exists at the right address.
The short version worth keeping: llms.txt is real, checkable, and used today, by a narrower set of systems than the conversation around it suggests. Google Search ignores it by its own admission, but developer tools, AI agents, and documentation platforms already treat it as useful, with AI-driven traffic to hosted docs rising from about 9% to 41-42% in under a year. Whether that's worth your time depends entirely on how much technical content your site carries, not on hype in either direction. Treat every claim in this space, including the ones made here, as worth checking against the primary source rather than accepted on repetition.
Frequently Asked Questions About llms.txt
Does llms.txt actually work?
It depends on what "work" means. If the question is whether it improves Google rankings, the answer is no, Google's own documentation, updated June 29, 2026, states that Google Search ignores files like llms.txt entirely. If the question is whether developer tools and AI agents can read it, the answer is yes, Mintlify, Anthropic, and other developer-facing platforms already do. There's no confirmed effect on AI Overviews or chatbot citations either way.
What's the difference between llms.txt and robots.txt?
robots.txt is a permissions file: it tells crawlers what they may or may not fetch, using Allow and Disallow directives tied to specific user agents like GPTBot or Googlebot. llms.txt has no permissions syntax at all, it's a curated content map written for AI systems to read, listing the pages worth prioritizing. Blocking a crawler and listing a page for AI systems are two separate jobs handled by two separate files.
How do you generate an llms.txt file?
List the pages that best represent your site (documentation, API reference, pricing, core product pages), write them into a Markdown file following the official structure (an H1 with your site name, a blockquote summary, and H2 sections of links), then publish that file at the root of your domain so it's reachable at yourdomain.com/llms.txt. The full walkthrough, including platform-specific hosting instructions, is covered step by step in how to create an llms.txt file.
Do ChatGPT, Google AI Overviews, and other AI tools actually use llms.txt?
No major AI provider has publicly confirmed using llms.txt to power citations or answers. Google's documentation says Search ignores it, and John Mueller has compared it to the old keywords meta tag, a field with no confirmed effect. The confirmed use case today sits with developer tooling: platforms like Mintlify auto-generate the file, and companies such as Anthropic serve their own, but that's a different audience than the chatbots most people are asking about.
Should a small business website bother with llms.txt?
That depends on whether there's real technical or documentation content behind the marketing pages. A small business site limited to a home page, service pages, and a contact form has little to gain from an llms.txt file: Google Search ignores it, and no AI provider has confirmed a citation benefit for that kind of content. A small business that also runs a knowledge base, an API, or extensive documentation sits closer to the audience the format was built for, where AI-driven traffic to hosted docs rose from about 9% to 41-42% between January and December 2025.
Blue handles your SEO and your GEO. On autopilot.
You approve, she produces content optimized for Google + AI.
Join the Lab · 5-day trial