The llms.txt File — Does It Matter for AI Discovery?

What llms.txt is, how AI tools use it, how to implement it on your website, and whether it is worth your time as an SEO signal.

What llms.txt is

llms.txt is a proposed standard created by developer Matt Webb. It is a plain-text file placed at the root of a website that tells AI tools and large language models what content is available and how to access it. Think of it as a robots.txt for the AI era: a simple, machine-readable way to say "here is the content you should use to answer questions about this organisation."

The file lists a title, summary and key links that the site owner wants AI systems to know about. It is designed to be consumed by AI tools that scrape and summarise web content for generative search, chat interfaces and knowledge-base bots.

How AI tools use it

Current AI systems such as ChatGPT, Claude and Google's AI Overviews rely on web crawling, similar to traditional search engines, to gather information. When an AI tool discovers an llms.txt file, it can use the listed links as direct sources for generating answers about that site or topic.

The file gives site owners a degree of editorial control over what AI tools surface. Without it, the AI decides which pages on your site are most relevant based on its own crawling and ranking logic. With it, you can specify the most important pages, the most accurate descriptions and the correct context for your content.

Implementing it on your site

Implementing llms.txt is straightforward. Create a plain-text file named llms.txt and place it in the root directory of your website. The format begins with a title line prefixed by # , an optional summary, and then a list of links with descriptions. Each link line starts with a dash, followed by the URL and a brief description.

A basic example for a business website might list the homepage, the about page, services pages and the blog archive. The file should be updated whenever you publish important new content that you want AI tools to discover. There is no official validation tool yet, but the format is simple enough to verify by eye.

What to include

Include pages that represent the best and most accurate version of your content. For a business, this typically includes your homepage, about page, key service or product pages, your most comprehensive blog posts and your contact page. Each link should have a description that helps the AI understand what the page contains.

Avoid including every page. The purpose is to curate, not to dump your sitemap. Think of it as a briefing document for an AI that wants to learn about your company. What five to twenty pages would you want it to read? Those are the pages that go in llms.txt.

Limitations

llms.txt is a proposed standard, not an official one. Adoption by AI tool providers is voluntary. While several AI tools and frameworks have added support, there is no guarantee that every AI system respects the file. It is also not a ranking signal for traditional search engines such as Google.

The file cannot prevent AI tools from accessing other pages on your site. If you want to block AI crawling entirely, you still need robots.txt and the appropriate meta tags. llms.txt is additive, not restrictive. It suggests what to use, not what to exclude.

Other AI discovery signals

llms.txt is one piece of a broader AI discovery strategy. Structured data such as schema markup helps AI tools understand the entities on your pages and their relationships. A clear site architecture with descriptive, unique page titles and headings improves how AI tools interpret your content.

Generative engine optimisation (GEO) is an emerging practice focused on making content perform well in AI-generated answers. While llms.txt suggests pages, GEO optimises the content itself to be cited. Read our guide on how to rank in AI search for a broader strategy. Also see our introduction to what GEO is for a definition of the practice.