llms.txt: What the Evidence Actually Shows, and What I Measured

Most writing about llms.txt assumes the argument is settled and the file won. The measurements do not support that.

In a study published in June 2026, Ahrefs analysed 137,210 domains — every site in its web analytics product that received traffic in May 2026 — using actual server-side request logs rather than surveys. Roughly 38,000 of those sites served a valid /llms.txt. Of those, 97% received no requests for the file at all that month. No AI bot, no crawler, no human. The study’s own wording: “No bots. No humans. Nothing.”

This site publishes an llms.txt. So this article is going to spend most of its length explaining why that file is probably doing nothing for me — and then report the one part of the specification I could actually test on my own setup, because that result was more useful than the theory.

What the proposal actually asks for

The spec lives at llmstxt.org and is at version 2, revised August 2026. It is a proposal, not a standard: no standards body ratified it.

The format is stricter than most implementations realise. In order:

• an optional byte-order mark
• an H1 with the project or site name — the only required section
• a blockquote with a short summary
• zero or more non-heading Markdown sections
• zero or more H2-delimited “file lists”, each a Markdown list of [name](url) with optional notes after a colon

The file goes at /llms.txt, or at any subpath, covering the URLs beneath it. Where several apply, an agent should use the most specific one. There is a conventional ## Optional section for links an agent can skip when context is tight.

The v2 additions are what most guides still miss. The proposal now asks sites to publish Markdown twins of their pages — at the same URL with .md appended (page.html.md) or substituted (page.md), and index.html.md / index.md for filename-less URLs. To let clients find them, it recommends standard link relations: rel="alternate" type="text/markdown" pointing at the Markdown version, and rel="describedby" pointing at the llms.txt that covers the page. Those can be HTML <link> elements or an HTTP Link: response header — and the header form, the spec notes, “can be added in web server or CDN configuration without modifying any pages.”

Two design questions v2 answers directly, and both are worth knowing. Why not /.well-known/ under RFC 8615? Because well-known URIs exist only at the origin root, and many authors control only a path: a GitHub Pages project site can publish in its own directory but can never write to the host’s /.well-known/. Why is sitemap.xml not a substitute? Because it usually omits Markdown versions, cannot point at useful external URLs, and in aggregate covers more than fits in a context window.

The mistake I made in my own file

My llms.txt had three lines of #-prefixed maintenance comments at the top, in Chinese, noting that the file was for AI search engines.

In Markdown those are H1 headings. So the “H1 with the name of the project” — the single required element — was actually the fourth heading in the file, and the three before it broke the required H1-then-blockquote sequence. Any parser following the spec would have read my maintenance note as the site’s name.

Nobody told me. There is no validator that warns you, no error surface, nothing in a log. The file returned HTTP 200 and looked fine to a human.

That is the first crack in the “it costs nothing” argument. Writing the file costs nothing; writing it wrong also costs nothing, which means you can spend an afternoon believing you shipped something you didn’t. I rewrote it to start at the H1, and the live version now begins where the spec says it should.

What the logs say

The Ahrefs dataset is the strongest evidence available because it is request logs, not opinions. The composition of the 3% that did get traffic is more instructive than the headline:

Who fetched it	Share of requests
SEO audit tools	21.7%
Unidentified / other	14.9%
General web crawlers (incl. Googlebot)	13.1%
Tech-profiling tools	11.6%
AI bots — all four categories combined	19.5%

Note that each of the top four categories individually out-fetches any single AI bot category. Ahrefs discloses that its own crawlers account for 10.6 of the 21.7 points above, leaving third-party SEO tools on 11.1%.

Within the AI 19.5%, the split matters more than the total: AI agents 10.5%, training crawlers 5.3%, assistants ~2.5%, retrieval bots 1.1%. Those retrieval bots — the class that fetches a page to answer a live user query inside ChatGPT or Perplexity — made 233 requests across the entire study. Claude Code, a coding agent, out-fetched every AI retrieval bot, every assistant and every training crawler. Slackbot alone requested llms.txt files more often than PerplexityBot did.

And 12.1% of requests came from tools that audit, score, validate or study the format — more than triple the combined traffic from retrieval bots and assistants. Ahrefs’ own summary: “A whole ecosystem has formed around auditing, scoring, validating, and studying the llms.txt standard, before we’ve even established whether any major AI platform actually reads it.”

The single most damaging finding is a negative one. /llms.txt requests that returned 404 received no AI traffic at all. Nothing is probing for the file. A file that nothing looks for cannot function as a discovery mechanism — it can only be found by something already crawling you, which by definition has already found you.

Ahrefs is careful about the limits: a fetch is not proof of a read, so every figure is “a ceiling on actual llms.txt consumption.” The sample is also biased toward technical sites, making the 28% adoption rate an upper bound — and making the 97% a floor on how bad the picture is for the group most motivated to make the file work.

Two independent checks point the same way. SE Ranking’s analysis of 300,000 domains found no connection between publishing llms.txt and AI citation frequency. Adoption itself is low: Rankability measured 5.6% of the Tranco top 10,000 serving a verified file, and an independent HTTP Archive analysis put the same figure at 5.61%.

What Google actually said

John Mueller has been consistent for over a year: llms.txt is “not done for search,” and he has described it as a “temporary crutch, perhaps to save some tokens” for AI coding tools. Google Search Central’s documentation states plainly that Search does not use it and that it neither helps nor harms rankings.

His structural argument is the part worth internalising. He characterised the file as telling systems “I have the best website ever, and here are all of the pages that everyone must go to.” Self-declared importance is worthless as a signal precisely because every site would claim it — which is why he compared it to the keywords meta tag.

The apparent contradiction people point to is Chrome’s Lighthouse audit, which added an llms.txt check in May 2026 and reignited the whole debate. Read the detail and it dissolves. If you have no file, the audit is marked not applicable, because the Lighthouse docs call the file optional “at the moment.” Only a file that exists and errors gets flagged. It is a hygiene check, not a ranking factor — and it generated about 22 requests across the entire Ahrefs dataset, roughly one in a thousand.

Lighthouse and Search are answering different questions. Lighthouse asks whether an agent that has already arrived finds the site convenient. Search asks whether to mention you to someone who has never heard of you. If your revenue depends on an agent completing a task on your site, the first question matters. If your goal is being quoted in an answer, it does not.

One inference trap to avoid: OpenAI, Anthropic and Google all publish llms.txt files for their own developer documentation, and this is cited constantly as platform support. Publishing a machine-readable index of your own docs says nothing about whether your crawler consults other people’s. Different teams, different problems.

The one case that holds up

Developer documentation is the context where llms.txt has real traction and a real payoff. When a developer asks a coding agent to integrate your API, the agent needs a compact, accurate map of your docs — and a Markdown index is genuinely cheaper to consume than 400 rendered HTML pages. Mintlify generates these for hosted docs sites; Anthropic, Cursor and Cloudflare serve them on their docs domains.

The pattern, stated precisely, is this: llms.txt helps an agent that has already decided to work with your site. It does not help an engine decide to mention your site to someone who has never heard of you.

That line sorts the whole question. Developer tools, APIs, SDKs, open-source projects: publish one and keep it current. Editorial sites, local businesses, e-commerce catalogues: close to zero return today.

What I measured: the Link header on Cloudflare Pages

The v2 claim that a Link: header can be added at the CDN layer without touching any page is checkable, and I checked it. This site runs on Cloudflare Pages, which reads a _headers file from the build output.

I added one line to the /* rule:

Link: </llms.txt>; rel="describedby"

Deployed, then read the live response headers. It works. The header is served on the homepage and on the file itself, and all four pre-existing security headers survived untouched — confirming again that _headers rules are additive rather than overriding, which is the opposite of what several third-party guides claim.

There is a real limitation, though. /* matches every response, so the favicon, the stylesheet and the JS bundle now all carry a header saying “the document at /llms.txt describes this resource.” That is meaningless for a binary. _headers cannot match on content type, so doing this precisely means enumerating paths by hand.

I also did not implement the other half of v2. rel="alternate" type="text/markdown" requires Markdown twins of every page, and this site has none. Generating them is real work — an output route per page, a build step, a second thing to keep in sync — and against the evidence above, the return is zero. So I have half of a proposal implemented, on purpose, with the useless half skipped.

I am keeping the line. It is one line of configuration and it is spec-compliant. But I am not going to describe it as an optimisation.

If you publish one anyway

• Start at the H1. No comments, no badges, no preamble above it, and a blockquote directly beneath. Check this by reading your own file as a parser would.
• Make the links worth following. They should point at machine-friendly content, not another layer of HTML.
• Do not let it rot. A hand-written llms.txt becomes a list of 404s the day you rename a URL — a file meant to help machines find your content, full of dead links. Generate it from the same source as your sitemap, or add its link check to whatever you already run over the site.
• Treat auto-generated files as untrusted input to agents. Ahrefs found a crawler studying llms.txt as a prompt-injection vector, precisely because agents trust content they ingest. If a plugin generates yours, read what it writes. You are handing an instruction to something that will act on it.
• Say only what your pages back up. The meta-keywords comparison lands because this file is you describing yourself; anything it links to needs to match the real page.

When to skip it

For most content sites the honest answer is: skip it, and spend the hour elsewhere. The specific place to spend it is a JavaScript check. Open your key pages with JavaScript disabled. If the content is not there, most AI crawlers cannot see it either — they do not render JS. That problem costs you far more citations than a missing Markdown index, and it is the one people skip while debating a file that 97% of the time nobody requests.

Where this leaves me

I keep llms.txt on this site for two reasons, and neither is AI visibility. It costs one file and one format check. And writing it forces me to state in a single paragraph what this site is about and who it is for — a paragraph that then earns its keep as the About page opening and the meta description, where it does measurable work.

The useful thing to take from the evidence is not that the format is bad. It is that the file’s readers are coding agents and crawlers building training corpora, not the retrieval pipelines that decide who gets cited. If that is your audience — if you sell software with an API — publish one today. If you are trying to get quoted in an answer, the levers are elsewhere.