Rawi MD — Markdown for AI Crawlers

Description

People are asking ChatGPT, Claude and Perplexity the questions they used to type into Google. To answer, those tools visit real websites and read them — including yours.

But they read a WordPress page badly. Your content arrives buried in markup, scripts, navigation and cookie banners that mean nothing to a language model, and the model pays for every one of those tokens.

This plugin answers Accept: text/markdown with a clean Markdown document at the same URL. Browsers keep getting the normal page. Nothing changes for your visitors, and nothing changes in how you write.

The part most plugins get wrong

Page caches answer requests before WordPress runs. When LiteSpeed or WP Rocket has a stored copy of a page, an agent asking for Markdown gets HTML instead, and the plugin never even loads. This plugin handles that on both sides: it tells the cache plugins to stand down at runtime, and it generates the server-level rules you need to paste in so the cache steps aside before PHP starts.

How agents can reach the Markdown

  • Accept: text/markdown on the normal page URL — real HTTP content negotiation, the route AI agents actually use
  • A .md suffix on any URL, for agents that do not send the header
  • ?format=md or ?output_format=md

The HTML page also advertises the Markdown version through a <link rel="alternate"> tag and a Link: response header, so an agent can find it without guessing.

What you control

  • Which post types are served, and which individual pages are excluded
  • Which theme elements are stripped, by CSS selector
  • Which YAML frontmatter fields appear
  • WooCommerce products carry SKU, price, currency, stock status and rating in the frontmatter. Those live in post meta, so a converter working from rendered HTML cannot produce them
  • A Content-Signal header stating how you allow the content to be used

Reporting

A dashboard shows which agents requested Markdown, which pages they wanted, and which route they used.

Privacy

The plugin records Markdown requests in a table in your own database: timestamp, post ID, URL path, user agent, and a one-way hash of the IP address. Nothing is sent anywhere. IP addresses are never stored in readable form, hashing can be turned off entirely, and entries are deleted after a retention period you set. Uninstalling removes the table.

Languages

The interface ships in English and Arabic. WordPress picks the Arabic strings automatically when the site language is set to Arabic, and the admin screens follow the right-to-left layout.

Screenshots

Installation

  1. Upload the rawi-md folder to /wp-content/plugins/.
  2. Activate the plugin.
  3. Open Settings > Rawi MD and pick your post types.
  4. Open the Server rules tab and apply the rules for your stack.
  5. Run the curl check from that same tab.

FAQ

Is this cloaking?

No. Cloaking means serving different content based on who is asking. This is content negotiation: the client states which format it prefers and the server honours it. The words are the same either way. The plugin never inspects the User-Agent to decide what to send.

That said, search engines have not published a definitive position on serving Markdown through content negotiation. The plugin sends noindex only on the .md and query-string addresses, which are genuine duplicates, and never on your real page URL.

I already use Cloudflare’s Markdown for Agents. Do I need this?

Probably not, if your zone is on a Pro plan or above and you are happy with the generic conversion. Running both means converting the same page twice. This plugin is for sites on the Cloudflare free plan, sites not behind Cloudflare, and anyone who wants control over what gets stripped and what metadata is attached.

Will this slow my site down?

No. Conversion only runs when a client explicitly asks for Markdown, and the result is cached on disk until the post changes. Normal visitors never touch this code path.

Does it work with Elementor?

Yes. Elementor renders through the content filters, which is the default source. Bricks and Oxygen replace the whole template, so switch the content source to the full page render for those.

How do I check it is working?

curl -sI -H “Accept: text/markdown” “https://example.com/your-post/”

You want content-type: text/markdown and an x-markdown-tokens header. If you get text/html, your page cache is answering first — see the Server rules tab.

Reviews

There are no reviews for this plugin.

Contributors & Developers

“Rawi MD — Markdown for AI Crawlers” is open source software. The following people have contributed to this plugin.

Contributors

Changelog

1.2.0

  • Renamed to Rawi MD. The previous name overlapped with an existing third-party feature name.
  • Every function, class, constant, option, hook and database table now uses the RAWI_ prefix.

1.1.1

  • Uninstall now removes cached files through WP_Filesystem rather than direct PHP calls.
  • Every database query passes the table name through a prepared identifier placeholder.
  • Minimum WordPress raised to 6.2, which is where identifier placeholders arrived.

1.1.0

  • Full Arabic translation, with a right-to-left admin layout.
  • Moved to its own top-level menu instead of sitting under Settings.
  • Rebuilt the Reports screen: headline figures, a 30-day activity chart, and proportional breakdowns by agent, page and route.

1.0.3

  • Product frontmatter now carries the main image URL, read from meta so it survives either content source.

1.0.2

  • Themes that print the title more than once no longer produce repeated H1 headings.
  • Default removal list now covers WooCommerce price, meta, tabs and related-product blocks.

1.0.1

  • Fixed a parse error that stopped the plugin activating.
  • The plugin now stays out of the way, and says so in the admin, when PHP is built without the DOM extension.
  • Conversion failures fall back to the normal HTML page instead of serving an empty document.

1.0.0

  • First release.