llms.txt

Jeremy Howard of Answer.AI proposed it in 2024. The file opens with the site's name as a heading and a one-line summary, followed by sections of links with short notes, ideally pointing to clean Markdown versions of the pages. It targets AI agents and coding assistants that need concise context rather than pages full of navigation, adverts and scripts. Some sites also publish an llms-full.txt with the complete text.

Support is uneven. It is most common on software documentation, including the developer docs of several AI companies, and Chrome's Lighthouse checks for it in its Agentic Browsing audits, but Google says its Search, AI Overviews included, ignores the file. It does not control access either; that remains the job of robots.txt.