llms.txt is a proposed standard: a markdown file at the root of your domain that gives language models a curated map of your most important content. The idea is reasonable and the format is genuinely well designed. The honest answer to whether you need one in 2026 is more complicated than most articles on the subject admit.
The short version is that no major AI search provider has documented consuming it as a ranking or retrieval signal, and Google has stated plainly that it does not use it. That does not make the file worthless, but it does mean the common advice to add one for AI search visibility is not supported by what the providers actually say. This guide separates the two cases.
Key takeaways
- llms.txt is a markdown index of your key pages, served at the root of the domain.
- Google has publicly stated it does not support llms.txt and has no plans to.
- No major assistant documents using it as a retrieval signal for AI search answers.
- It has a real and growing role in agent and developer-documentation contexts.
- It costs very little to publish, so the case against it is opportunity cost, not risk.
What the file actually is
The proposal is a plain markdown file served at the root of a domain, listing the pages you consider most important with a short description of each. Some sites also publish an expanded variant that inlines the full text of those pages so a model can read the content without following links.
The motivation is sound. A model working from raw HTML has to strip navigation, cookie banners, and promotional furniture before reaching the substance, and it has no reliable way to know which of your thousands of URLs you consider canonical. A curated markdown index solves both problems cleanly, and the format is deliberately simple to produce.
It is worth being precise about what it is not. It is not a robots directive and does not control crawling or grant permission. It does not submit anything. It is a hint, and hints only work when the recipient has agreed to read them.
What the providers have actually said
This is where most coverage of the topic becomes unreliable. Google has stated publicly that it does not support llms.txt and is not planning to, and its guidance on optimizing for AI features tells site owners the file is not needed for AI Overviews, AI Mode, or other generative search surfaces. A Google search advocate compared it to the keywords meta tag, noting that bots were not requesting the file.
Across the other major providers the position is mostly silence rather than rejection. No major vendor has documented that its production retrieval systems consume llms.txt from arbitrary external websites as an input to answering user questions. There are reports of the files being fetched by some crawlers, but a fetch is not evidence of use as a ranking signal.
The counterweight is that the file has found genuine traction in agent and developer-tooling contexts, where documentation is being read deliberately by a model working on a task rather than retrieved to answer a search query. That is a real use case with real adoption, and it is a different use case from AI search visibility.
- Google: publicly stated non-support for AI Overviews and AI Mode
- Other major assistants: no documented use as an external retrieval signal
- Agent and developer-docs tooling: meaningful and growing adoption
- Observed crawler fetches: real, but not evidence of ranking influence
- Adoption across the web: still a small minority of sites
So should you publish one
For most sites the answer is yes, but for the right reason and with the right expectations. The file takes very little effort to produce, carries no ranking risk, and positions you for a standard that may gain support later. What it will not do is improve your visibility in AI search answers today, and any tool selling it on that basis is overstating the evidence.
The case is strongest if you publish documentation, an API reference, or any material that developers and coding agents consume directly. In that context models genuinely do read curated markdown, and a good llms.txt measurably improves how well an agent understands your product. We publish one at the root of this site for exactly that reason.
The case is weakest if you are a local business or a content site hoping it will get you into AI Overviews. It will not, and the hours are better spent on the things that demonstrably influence citation, which our guide to getting cited by AI covers in detail.
How to write one that is actually useful
Start with a heading naming the site, then one blockquote summarising what the organisation does in a sentence. Follow with grouped lists of links, each with a short description explaining what the reader will find. Keep it curated rather than exhaustive: the value is in the selection, and a file listing every URL recreates the problem it was meant to solve.
Prioritise pages that answer questions rather than pages that convert. A pricing page and a product overview belong in it, but so do your best explanatory guides, because those are what a model needs when reasoning about your category.
Keep it current. A stale index pointing at moved or deleted pages is worse than no index, since it actively misleads anything that does read it. If you maintain a sitemap generation step already, regenerate this alongside it.
Where the effort actually pays instead
If the goal is AI search visibility rather than agent readability, the evidence points elsewhere. Being retrievable and competitive for the query comes first, because assistants overwhelmingly draw from pages that already rank. Structuring answers so they can be lifted cleanly comes second. Corroboration across independent sources comes third.
Structured data is the closer analogue to what people hope llms.txt will do, and unlike llms.txt it is documented, consumed, and acted upon by search systems today. Our guide to schema markup for AI search covers which types are worth implementing.
Publish the file, spend twenty minutes on it, and move on to the work that moves the number. If you want a prioritised view of which of these gaps applies to your own site, run an audit rather than working from a generic checklist.
FAQ
Questions about this guide
Does Google use llms.txt?
No. Google has stated publicly that it does not support the file and is not planning to, and its own guidance on AI features tells site owners it is not needed for AI Overviews or AI Mode.
Will llms.txt hurt my SEO?
No. It is an additional file that traditional search systems ignore, so there is no ranking downside. The only real cost is the time spent producing and maintaining it.
What is the difference between llms.txt and robots.txt?
robots.txt is a widely honoured directive controlling what crawlers may access. llms.txt is an optional content hint with no access-control function and no committed support from major providers. They solve unrelated problems, and our robots.txt checker will tell you what your directives actually permit today.
Who should definitely publish one?
Sites with developer documentation, API references, or technical material that coding agents read directly. Adoption is real in that context, and a curated markdown index measurably improves how well an agent understands the product.