llms.txt

TL;DR. llms.txt is a Markdown file sitting at a domain's root, at yourdomain.com/llms.txt, meant to hand AI systems a curated summary of a site's structure instead of raw, ad-and-navigation-heavy HTML. Jeremy Howard of Answer.AI proposed the format in September 2024, and by 2026 SE Ranking found adoption near 10% across 300,000 domains studied, though Search Engine Journal's analysis of Ahrefs data found 97% of published llms.txt files receive zero requests, leaving the format's actual effect on AI citations a genuinely open question rather than a settled best practice.

What is llms.txt?

llms.txt is a plain Markdown file living at a website's root, /llms.txt, giving a language model a clean, curated outline of a site's structure and most important pages, similar in spirit to a sitemap but written for a model to read rather than a crawler to parse. The official specification at llmstxt.org defines two files: /llms.txt, a compact navigation summary, and /llms-full.txt, a comprehensive version containing fuller documentation content in one place.

Key highlights

• Jeremy Howard, co-founder of Answer.AI and fast.ai, proposed llms.txt in September 2024 as a documentation-first standard.
• SE Ranking's study of 300,000 domains found a 10.13% adoption rate after roughly eighteen months of industry attention.
• Search Engine Journal's analysis of Ahrefs data found 97% of published llms.txt files received zero requests.
• Trakkr's study of more than 337,000 citations found zero statistical correlation between having an llms.txt file and AI citation rates, a p-value of 0.85.
• AI retrieval bots tied to ChatGPT and Perplexity accounted for only about 1% of requests made to llms.txt files, while Claude-Code and GPTBot were the top individual bots actually requesting them.

llms.txt vs robots.txt

llms.txt and robots.txt differ in purpose, not just format. robots.txt tells a crawler what it is allowed to access. llms.txt tells a model what a site is about, a curated summary rather than an access rule, and it carries no enforcement mechanism whatsoever, since nothing requires a model to read it, let alone respect it.

Does llms.txt actually help with AI citations?

Evidence on llms.txt's citation effect is genuinely mixed. Search Engine Land tracked 10 sites directly and found no clear citation lift tied to publishing the file, while Trakkr's larger citation study turned up zero measurable correlation. Google's John Mueller has publicly called llms.txt a temporary crutch, perhaps to save some tokens, aimed more at AI coding tools than AI search. At the same time, Anthropic serves its own documentation through llms.txt, and Claude-Code shows up as one of the heaviest actual requesters of the file, which points to real utility for agentic coding and documentation retrieval even where evidence for a general AI-search citation boost stays weak.

Creating an llms.txt file

1. Start with an H1 site name, a one-line summary blockquote, and a short list of the pages most important for a model to understand the site.

1. Group links under H2 sections, Documentation, Guides, Examples, so a model can navigate by category instead of scanning one flat list.
2. Keep the file to genuinely essential pages, since the entire value proposition is a curated summary, not a full sitemap dump.
3. Publish a companion /llms-full.txt only if the site has substantial documentation worth including in full, since most sites do not need the second file at all.
4. Treat llms.txt as a low-cost, low-priority addition rather than a primary GEO lever given the mixed evidence, and put crawlability and extractability work first.

Frequently asked questions

What is llms.txt?

llms.txt is a Markdown file placed at a website's root that gives AI systems a curated summary of the site's structure and most important pages, functioning like a sitemap written for a language model instead of a search crawler.

Does having an llms.txt file actually move AI citation rates?

The evidence is weak at best. Trakkr's study of more than 337,000 citations found zero statistical correlation, and Search Engine Land's tracking of 10 sites found no clear citation lift either, though the file does show genuine usage among agentic coding tools like Claude-Code.

Who came up with the llms.txt standard, and when?

Jeremy Howard, co-founder of Answer.AI and fast.ai, proposed llms.txt in September 2024, pitched as a way to give language models a documentation-friendly summary of a website.

Should every site bother publishing one?

Not as a priority. Search Engine Journal's analysis found 97% of published files get zero requests, so it is reasonable to add as a low-cost extra, particularly on documentation-heavy sites, but it should not come ahead of crawlability and extractability fixes that have actual proven impact.