Generate llms.txt

[souzatharsis]

Generate llms.txt for the Book

We propose that those interested in providing LLM-friendly content add a /llms.txt file to their site. This is a markdown file that provides brief background information and guidance, along with links to markdown files (which can also link to external sites) providing more detailed information.

Useful for downstream LLM-based consumption.

[impredicative]

LLMs are a transient phenomenon. They will eventually be replaced by more general models, e.g. multimodal world models, etc. For this reason, I don't think it's wise to settle on llms.txt.

[souzatharsis]

We can wait for the future or act now.
Nothing is settled but only transiently accomplished.

[impredicative]

I favor extending robots.txt instead.

If I request llms.txt, the server will think I am an LLM, and maybe block my IP at once if the server is hostile to LLMs. I could then be worse off.

Instead, if I read robots.txt, the server will think I am an abiding crawler.