What is llms.txt, and do you actually need one?

What is llms.txt? A plain-English definition

llms.txt is a proposed convention for helping AI systems understand a website. So what is llms.txt in practice? It is a single plain-text file, written in markdown, placed at the root of your domain so it can be found at a predictable address: yoursite.co.uk/llms.txt. Inside, it gives a short summary of what the site is, followed by a curated list of links to the pages that matter most, each with a one-line description.

The idea was proposed in September 2024 by Jeremy Howard, co-founder of the AI research lab Answer.AI, and the specification lives at llmstxt.org. The reasoning behind it is straightforward: large language models work with limited context windows, and a typical web page is a poor use of that space. Navigation menus, cookie banners, scripts, and footers all get in the way of the actual content. A hand-written markdown summary strips all of that out and hands an AI system the essentials in a format it parses easily.

Think of it as the difference between giving a visitor a map of the building versus letting them wander the corridors. The corridors still exist, and search engines still walk them. llms.txt is just the map pinned up at the entrance, in case anything reading on a visitor's behalf wants one.

How the proposed llms.txt standard works

The format is deliberately minimal, and that minimalism is the point: anyone who can write a bulleted list can write one. According to the specification, a valid llms.txt file contains, in order:

• An H1 heading with the name of the site or project. This is the only required element.
• A blockquote summary: one or two sentences describing what the site is and who it is for.
• Optional detail paragraphs giving any context an AI system would need to interpret the links that follow.
• Sections of links, each under an H2 heading such as "Services" or "Guides", where every link has the format: name, URL, and a short note about what the page contains.
• An "Optional" section at the end, listing secondary pages an AI system can skip when context space is tight.

The proposal also describes a companion pattern: publishing markdown versions of individual pages by appending .md to their URLs, and a larger llms-full.txt file that inlines the full text of key pages rather than just linking to them. Some documentation platforms have adopted these variants, because technical documentation is the use case where the idea has found the most genuine traction: developers ask AI assistants questions about tools, and the tools' makers want the answers to come from their own docs.

For a service business website, the file is shorter and simpler. Ours, at freshandcrispy.co.uk/llms.txt, summarises what Fresh & Crispy does, then lists the service pages, the audit, pricing, and the key guides, each with a one-line description. Writing it took less time than this article.

llms.txt vs robots.txt vs sitemap.xml: three files, three jobs

The name invites confusion with two files that have been around for decades, so it is worth being precise about who does what. All three live at (or are referenced from) the root of your domain, and that is roughly where the similarity ends.

File	Job	Status
robots.txt	Tells crawlers which parts of the site they may or may not fetch. A permissions file: it restricts access.	Long-established convention, formally documented by Google's robots.txt guidance and respected by mainstream crawlers.
sitemap.xml	Lists the URLs on the site so search engines can discover and recrawl them efficiently. A discovery file: it announces everything.	Long-established standard, documented in Google's sitemap documentation and consumed by every major search engine.
llms.txt	Summarises the site and points AI systems at a curated selection of its best content. An editorial file: it prioritises.	A 2024 proposal with uneven adoption. No major AI provider has publicly confirmed that its crawlers read or use it.

The distinction that matters most: robots.txt and sitemap.xml are load-bearing. Get robots.txt wrong and you can block your entire site from search; skip a sitemap on a large site and pages get discovered slowly or not at all. llms.txt, in its current state, is decorative by comparison. Nothing breaks without it, and nothing is confirmed to change with it.

The honest status: a proposal, not a standard

This is the part most articles about llms.txt soften, so let us not. As of now, llms.txt is a proposal. It is not an agreed web standard, it has not been adopted by a standards body, and, most importantly, none of the major AI companies has publicly confirmed that its production systems fetch or use the file. Google's guidance on appearing in AI features does not mention llms.txt at all: it points back to ordinary indexing, crawlability, and helpful content. OpenAI, Anthropic, and Perplexity document their crawlers' behaviour around robots.txt, not llms.txt.

Adoption on the publishing side is real but lopsided. A number of developer-tool and documentation sites publish the file, some content platforms generate it automatically, and various directories track sites that have one. But publishing a file and having it consumed are different things, and evidence of the second is currently thin to nonexistent. Server log analyses shared publicly by various site owners have generally found little or no crawler traffic to the file from the major AI bots.

None of that makes the idea bad. Plenty of useful conventions started as informal proposals, and this one is well designed for a genuine problem. It simply means any claim that llms.txt will get you cited by ChatGPT, or ranked in AI Overviews, is ahead of the evidence, and you should hold your budget accordingly.

The case for adding an llms.txt file

Even with all those caveats, there is a reasonable argument for spending the hour. First, the cost is close to zero: one markdown file, no code changes, no design changes, no risk to your existing search visibility. Second, the act of writing it is useful in itself. Summarising your business in two sentences and choosing your ten most important pages is a clarity exercise, and if you find it difficult, that difficulty is telling you something about the site. Third, if the convention does get adopted by major crawlers later, sites that already publish a well-maintained file are ready on day one. Fourth, some smaller AI tools, agents, and scrapers do look for it now, and being legible to the long tail costs nothing extra.

There is also a quiet positioning benefit. Prospective clients and partners increasingly check how a business presents itself to AI systems. A tidy llms.txt sits alongside clean markup and fast pages as a small signal that the site is looked after.

The case against bothering with llms.txt

The argument against is equally simple: opportunity cost and false comfort. An hour spent on llms.txt is an hour not spent fixing the things that demonstrably affect AI visibility, which are the same things that affect search visibility: pages that load their content in the initial HTML, headings that describe what follows, and answers stated plainly near the top of the page. We covered what actually determines whether AI systems can use your site in our guide to AI search visibility, and llms.txt appears nowhere on that list because the systems doing the citing have not said they read it.

The false comfort risk is the bigger one. A business that adds llms.txt and considers its "AI strategy" handled has spent an hour to avoid the real work. If the file drifts out of date, listing services you no longer offer or prices that have changed, it could in principle even feed stale information to whichever tools do read it. A misleading map is worse than no map.

Our view, for what it is worth: add one if the rest of your house is in order, skip it if it is not. It is the last item on the list, not the first.

How to write an llms.txt file, step by step

If you decide to add one, here is the whole job. You need a text editor and access to upload one file to your web root.

• Start with the H1. One line: a hash symbol, a space, and your business name.
• Write the blockquote summary. One or two sentences beginning with "> " that say what you do, who for, and where. Plain words, no slogans. If an AI system only reads this far, it should still describe you accurately.
• Add link sections. An H2 per group ("## Services", "## Guides", "## Company"), then a bulleted list where each item is a markdown link followed by a colon and a one-line description of what is on that page.
• Curate hard. Ten to twenty links, not two hundred. This file is an editorial choice, not a second sitemap. Push anything marginal into a final "## Optional" section or leave it out.
• Upload it to your root so it resolves at yourdomain.co.uk/llms.txt, and check it loads as plain text in a browser.
• Put a review date in your calendar. Twice a year is enough for most service businesses. A stale file undermines the only benefit it has.

That is genuinely all there is to it. Anyone who quotes you a four-figure price for "llms.txt implementation" has told you everything you need to know about them.

What matters more than llms.txt for AI visibility

If your goal is being found and cited by AI systems, the ranked list of what to fix looks like this. First, make sure your content is crawlable and indexed at all, since a page no system can fetch can never be cited, whatever files sit at your root. Second, structure every important page so a single passage answers a specific question, because that is what answer engines quote. Third, keep your business facts, name, services, location, consistent everywhere they appear. We walk through the practical steps in how to get cited by ChatGPT and Perplexity, and the groundwork underneath all of it is ordinary, honest SEO work: the boring fundamentals did not stop mattering because the interface reading them changed.

If you are not sure where your own site stands on any of that, the free CRISP audit scores it in about a minute, and the structural checks it runs, clarity, crawlability, whether a page states its point, are exactly the ones that decide AI visibility long before a markdown file enters the picture.

Frequently asked questions