Most of the documentation I depend on, I no longer read first. An agent reads it. When I am debugging LiveKit worker behavior or a Twilio SIP setting, the sequence is that a coding agent fetches the relevant page, quotes the part that matters, and I go verify the part I do not believe. I still read docs. I just read them second, after something else has narrowed them down.
That changes what makes a documentation site good, and not in the direction most doc advice points. An agent does not scroll, does not recognize your navigation, does not learn your information architecture across visits, and does not get a tooltip. It gets a chunk of text and has to answer from it.
I first wrote this post in June 2025, and it was mostly assertion. It leaned on a couple of dramatic search-traffic statistics I could not properly source, and it quoted Andrej Karpathy from memory without a link to the actual post. I have cut all of that. What follows is what I could check myself, with the commands I checked it with, plus what I still cannot measure.
The thing you can do today that measurably helps
Serve a plain-markdown version of every page at a predictable URL.
Several documentation sites now do this by appending .md to the page URL. Checked on September 26, 2026:
curl -sL https://code.claude.com/docs/en/memory.md
curl -sL https://docs.livekit.io/agents.md
curl -sL https://docs.stripe.com/payments.md
curl -sL https://developers.cloudflare.com/agents/index.md
All four return text/markdown. The size difference is the point:
| Page | HTML | Markdown |
|---|---|---|
| code.claude.com/docs/en/memory | ~759 KB | ~54 KB |
| docs.livekit.io/agents | ~379 KB | ~10 KB |
Be careful reading those numbers. The HTML figure includes the application shell, inlined scripts, and framework payload, not just prose, so the ratio overstates how much text is saved. But the direction is real and the mechanism is obvious. Every byte of navigation markup, cookie banner, sidebar, and client-side hydration data that reaches the model is context spent on something that is not your answer, and worse, it is context the model might misread as content.
This is the change with the clearest cost-benefit case. It is cheap to add, it needs no cooperation from any AI vendor, and it works with a plain HTTP fetch, which is the one capability every agent has.
llms.txt, and what it is honestly worth
Jeremy Howard proposed llms.txt in September 2024, with the specification at llmstxt.org. The shape is a markdown file at your root: a title, a one-line summary, then curated sections of links with a sentence of description each. The idea is a map for a reader with a limited context window, not a full dump.
Adoption among developer-facing docs is real. Live as of today:
curl -sL https://docs.anthropic.com/llms.txt
curl -sL https://docs.livekit.io/llms.txt
curl -sL https://developers.cloudflare.com/llms.txt
curl -sL https://docs.stripe.com/llms.txt
curl -sL https://mintlify.com/docs/llms.txt
Now the honest part. llms.txt is a convention, not a standard, and unlike robots.txt it has no enforcement mechanism and no published commitment from any major crawler that it is consulted. So the defensible reason to ship one is not that it will be crawled, it is that it is the file you point an agent at, by URL, when you want it grounded in your docs.
There is at least one clear signal that vendors are designing for exactly that. Fetch code.claude.com/docs/en/memory.md today and the first three lines of the response are a pointer back to code.claude.com/docs/llms.txt, telling whatever is reading to fetch the index before exploring further. The doc is written on the assumption that its reader is an agent that arrived mid-site and needs a map.
This site ships two files. /llms.txt is the index, and /llms-full.txt is the long-form text of every public article and project in one document. Both are generated from the same frontmatter and MDX that render the pages, which is the part I would insist on: a hand-maintained machine file drifts from the human one within a month, and then you are shipping two versions of the truth. Anything I mark unlisted is excluded from both, so the machine view and the public view agree about what exists.
Structure for a reader that gets one chunk
Retrieval does not hand a model your page. It hands it a fragment, usually without knowing what came before it. Everything below follows from that.
One page, one question, one stable URL. The unit of retrieval is roughly a page or a section, so a page that answers three questions gets retrieved for all three and answers each one partially. Split them.
Put the answer in the first paragraph. Not the motivation, not the history of the feature, not what the reader will learn. The chunk that gets retrieved is usually the top of the page, and an agent will confidently answer from it whether or not the real answer was further down.
Make headings self-contained. "Configuration" tells a model nothing once it is detached from the page title. "Configuring SIP trunk failover" survives on its own.
Write examples that run. Complete imports, real parameter names, and a pinned version. Fragments with ... in the middle get reassembled by the model into something plausible and wrong, and the user blames your API. This is the single most common way I watch agents produce broken code against a real library.
Document what the thing does not do, and the exact error strings. Negative information is what stops an agent from inventing the method you never shipped. If a limit exists, state the number. If a call fails in a specific way, put the literal error text on the page, because that string is what somebody is pasting into a prompt.
Never break a URL. This mattered before, and it matters more now, because URLs are what end up quoted in a model's answer and cached in someone's agent config. An anchor is an address. Redirect everything, forever.
Doc servers, which cut out the guessing
The next step up from serving good text is serving it over a protocol the agent can query. The Model Context Protocol is the emerging shape for that, and a documentation server is one of its most obviously correct applications. Instead of an agent fetching a page and hoping it is the right one, it searches your docs and gets back the relevant sections.
I would not oversell this relative to the markdown endpoints. A doc server requires the user to have configured it; a .md URL requires nothing. Do the cheap one first. But if you maintain a large API surface, being queryable rather than crawlable is the difference between an agent finding your current endpoint and repeating one it memorized from a 2024 blog post.
What I cannot measure, and it bothers me
I do not know whether any of this improves the answers people get about my work. I cannot tell from my logs whether an agent that answered a question about voice agents read my llms-full.txt, read a page, or answered from training data that happens to include me. There is no attribution channel. Requests from AI clients arrive looking like requests, and the answer is assembled somewhere I cannot see.
That is a genuinely unsatisfying place to be, and it is why I am skeptical of confident claims in this area, including the ones I made in the first version of this post. Nobody optimizing docs for LLMs right now has a feedback loop worth the name.
So I have settled on doing the things that are defensible without measurement. Clean markdown at a predictable URL costs almost nothing and is strictly better than shipping an application shell to a text model. Answer-first pages and runnable examples are things I wanted anyway, for humans, and the machine case only makes them more urgent. Stable URLs were always the right call. An llms.txt is cheap and I get direct use out of it myself.
The advice I would resist is the reframing of documentation as a content-marketing channel aimed at models. The reason to write a precise page that states its limits and shows a complete example has not changed. The audience just got larger, less forgiving, and considerably worse at guessing what you meant.