Machine-readable sites: llms.txt and structure

Three separate things get conflated: a file describing your site for language models, your crawler policy, and the underlying structure of your pages. The third matters most.

Clean structure first

Machine intermediaries parse HTML. What helps them is what has always helped accessibility and search.

The last row catches many modern sites. Content that only appears after client-side rendering is invisible to a substantial share of crawlers.

The llms.txt convention

A markdown file at the root of the domain describing what the site contains and linking to the most useful pages, in a form a language model can read cheaply.

It is a proposed convention rather than an adopted standard, and support is uneven. It also costs perhaps an hour to produce.

A workable structure: the organisation in one paragraph, what it does, the main topics covered with links, and links to any structured data or documentation.

Keep it current. A stale file is worse than none.

The crawler decision

Robots directives can permit or block the crawlers used to build and serve these systems. There are two separate categories: crawlers that gather training data, and crawlers that fetch pages to answer a live query.

Blocking the second removes you from generated answers. For most businesses that is the wrong trade, since being cited is the objective.

Blocking the first is a defensible position for publishers whose content is their product. For a business using content to generate enquiries, it forgoes the benefit for no commercial gain.

Structured data

Schema markup states unambiguously what a page contains: organisation, product, service, price, opening hours, review, question and answer.

It is well established for search and directly useful to machine intermediaries. The types worth implementing for most businesses are organisation, local business, service or product, FAQ, and article with author and date.

The practical checklist

1. Content rendered server-side and readable without scripts.
2. Prices, specifications and contact details as text.
3. Correct heading hierarchy on every page.
4. Structured data for the relevant types.
5. Visible dates on content, updated when the content is.
6. A crawler policy that is deliberate rather than inherited.
7. An llms.txt file, kept current.
8. Consistent facts across your site, your listings and directories.

Items 1 to 5 do the work. Items 6 to 8 are cheap and worth doing.