Site Answers – llms.txt, AI Crawler Log & Markdown

Description

AI assistants are reading your website. Site Answers has one job: make your site
readable to them, show you whether they actually came, and let you decide which of them
may. These parts do that job, and together they are the whole plugin.

It publishes an llms.txt file. llms.txt is an emerging convention — a plain,
readable Markdown index at the root of your site that tells AI assistants what you
have published and where to find it. Site Answers builds yours from your own posts and
pages: your site name, your tagline, then one line per URL with its title and a short
description. It is rebuilt automatically whenever you publish, edit or delete
something, so it never goes stale.

It publishes llms-full.txt too. Where llms.txt is the map, llms-full.txt is the
whole territory: the full text of every page llms.txt lists, one after another, in one file,
so an AI assistant can read your site in a single visit. Same pages, same exclusions, capped at
5 MB with a note saying where the rest is. You can switch it off in Settings.

It logs which AI crawlers visit. GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot,
CCBot and the others are recognised by name. For each visit you see which crawler
came, which page it asked for, and whether it got the page or an error. That answers a
question site owners currently cannot answer at all: is AI search actually reading my
content, and which pages does it want?

It serves a clean Markdown copy of every page. Add .md to the end of any post or page
address and you get the article as plain Markdown — headings, paragraphs, lists, links and
tables — with the navigation, sidebars, share buttons and scripts left out. That is what an
automated reader wants, and it is far less likely to be misquoted than a page it had to dig out
of a theme. Each HTML page also advertises its Markdown twin in a Link header, so a crawler
can find it without guessing.

It tells you when your robots.txt is turning AI crawlers away. An empty crawler log often
has a simple cause: a line in robots.txt — added by another plugin, a host, or years ago by
hand — asking GPTBot or ClaudeBot to stay away. The Overview reads the robots.txt your site
actually serves, the way the crawlers read it, and names every AI crawler it blocks.

And it lets you choose, crawler by crawler. In Settings, each AI crawler has three
choices: No change, Allow or Block. Block ChatGPT’s training crawler and keep the one that
fetches your page when someone asks ChatGPT about you, or the other way round. Your choices go
into WordPress’s own robots.txt. Until you pick something, robots.txt stays exactly as it was.

How this is different

Plenty of plugins will write an llms.txt for you. Almost all of them stop at that point: the
file is written, and you never find out whether anything read it. This one closes the loop — it
publishes the file, serves your pages in a form a machine can actually read, and then shows you
whether any AI crawler turned up and what it asked for. Publishing without measuring is
guesswork, and until now measuring meant reading raw server logs.

Three further differences, each of which you can check in the code:

• It stands aside rather than fighting for the address. If another plugin, or a static file
  in your web root, already serves /llms.txt, Site Answers never registers its own route for it:
  theirs keeps working untouched, and your settings screen tells you which one is answering. Two
  plugins answering one address is how sites break, and it is always the newer plugin that gets
  blamed for it.
• It keeps nothing personal. The crawler log holds the crawler’s name, the page path, the
  response status and the time. No IP addresses, no query strings, no headers — so it never grows
  into a consent-banner problem the way a general-purpose visitor log does.
• It asks for nothing. No account, no API key, no external service, no outbound request of
  any kind, and nothing to upgrade to. Everything happens on your own server, so nothing about
  your site or your visitors leaves it.

What it does not do

This plugin does not connect to any external service. It makes no outbound HTTP request
of any kind — no API, no phone-home, no telemetry, no analytics, no third-party
libraries. Everything it shows you was produced by your own site, on your own server.

It stores no IP addresses. The crawler log records the crawler’s name, the page path and
the time — nothing else. Anything after the ? in an address is discarded before the
record is written, because query strings on real sites carry e-mail addresses, order
references and reset tokens. Records older than 90 days are deleted automatically.

It changes nothing on its own. Your robots.txt is left exactly as it is until you choose
Allow or Block for a crawler, and no file is ever written to your server: if you have a
robots.txt file of your own, the plugin tells you so and leaves it alone.

Why this matters

AI assistants increasingly answer questions using content they read from websites, and being
readable to them is becoming its own discipline — answer engine optimization, or AEO, alongside
the generative engine optimization people now talk about next to ordinary SEO. Both come down
to the same two practical questions: can a large language model find and parse what you
published, and is it actually doing so? Site Answers gives you a straight answer to the second
one, and does the groundwork for the first.

Who it is for

Anyone who publishes writing and wants to know whether AI search engines are reading it.
Bloggers, documentation sites, shops with real content, agencies who need to show a
client what is happening.

Privacy

No personal data of any kind is collected, stored or transmitted. Because no IP address
is recorded and nothing leaves your server, this plugin does not put your site into
consent-banner territory.