llms.txt
A proposed standard file that points AI engines to a site's key content.
llms.txt is a proposed convention — published at llmstxt.org by Jeremy Howard of Answer.AI on 3 September 2024 — for a markdown file served at a site's root that gives large language models a curated, concise index of the site's most important content. It addresses the fact that LLM context windows are too small to ingest full HTML pages with navigation, ads and scripts. The format requires only an H1 (the site name) plus optional blockquote summary and H2 link-list sections; a companion llms-full.txt can carry expanded content.
Importantly, llms.txt is a community proposal, not a ratified standard, and as of 2026 there is no public evidence that major engines (OpenAI, Google, Anthropic) consume it as a crawling or ranking directive. Adoption so far is concentrated among publishers who provide the file (e.g. Anthropic, Stripe, Cloudflare). It should therefore be treated as a low-cost, forward-looking signal that complements — not replaces — robots.txt, sitemaps and structured data.
Sources
- The /llms.txt file — specification — llmstxt.org (Jeremy Howard)
- /llms.txt — a proposal to help LLMs use websites — Answer.AI
- Meet llms.txt, a proposed standard for AI website content — Search Engine Land