llms.txt

It is a convention, not a standard

Worth saying plainly, because the distinction gets blurred in marketing copy. No standards body has ratified llms.txt. There is no conformance test and no registry. It is a proposal that several documentation platforms have adopted because it is useful, and adoption is what gives it whatever weight it has.

Treating it as a standard in front of a technical audience is an own goal. Treating it as a useful convention that your tooling implements is accurate and sufficient.

What the file contains

The shape is deliberately simple:

• An H1 with the site or product name
• A blockquote summarising what the site is, in a sentence or two
• H2 sections grouping links, each link followed by a short description of what it covers
• Optionally, a Notes section for anything that does not fit

The point is that an agent can read the whole file in one request and know what exists before fetching anything else. That is the entire value proposition: discovery in one round trip, in a format that does not require stripping navigation, cookie banners and footers out of HTML.

How it differs from robots.txt and sitemap.xml

The three are often grouped together and do unrelated jobs.

• robots.txt states permissions — which crawlers may fetch which paths. It is a policy file.
• sitemap.xml is an enumeration — a machine-readable list of every URL, built for search engine crawlers, with no description of what any of them contain.
• llms.txt is a curated index — a human-selected subset, with prose explaining what each item is and why it matters.

A sitemap tells a crawler where everything is. An llms.txt tells an agent what is worth reading. A site can and usually should have all three.

What it does not do

It is a signpost, not an interface.

An llms.txt file does not serve content — the agent still has to fetch each page it points to, and still has to deal with whatever format those pages come in. It does not let an agent search, filter, or ask a question. It carries no authentication and no concept of permissions, so it describes only what is already public.

For retrieval that goes beyond "here is the list," the agent needs an actual interface — which is what an MCP server provides. The two are complementary rather than alternative: the index aids discovery, the server handles retrieval.

Implementing it well

• Curate, do not dump. An llms.txt that lists every URL is a sitemap with extra steps. The descriptions are the value.
• Keep it current. A stale index is worse than none, because it sends agents to content that has moved.
• Pair it with clean content. An index pointing at pages only available as rendered HTML solves half the problem. Serving Markdown alongside solves the other half.
• Be honest about scope. If the index covers only part of the site, say so in the file.