Title: ADG Markdown for AI Crawlers
Author: andyodua
Published: August 27, 2026
Last modified: September 3, 2026
---
Search plugins
# ADG Markdown for AI Crawlers
By [andyodua](https://profiles.wordpress.org/andyodua/)
[Download](https://downloads.wordpress.org/plugin/adg-markdown-for-ai-crawlers.1.1.0.zip)
* [Details](https://wordpress.org/plugins/adg-markdown-for-ai-crawlers/#description)
* [Reviews](https://wordpress.org/plugins/adg-markdown-for-ai-crawlers/#reviews)
* [Installation](https://wordpress.org/plugins/adg-markdown-for-ai-crawlers/#installation)
* [Development](https://wordpress.org/plugins/adg-markdown-for-ai-crawlers/#developers)
[Support](https://wordpress.org/support/plugin/adg-markdown-for-ai-crawlers/)
## Description
When a known AI crawler (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and
others) requests one of your posts or pages, this plugin returns clean **Markdown**
of the main content on the **same URL** instead of the full theme HTML — no header,
footer, sidebar or scripts. Regular visitors keep seeing your normal site.
It also generates a virtual **llms.txt** and **llms-full.txt** — a sitemap-style
index of your content for AI tools — served at `/llms.txt` and `/llms-full.txt`
without creating any physical files.
**What it does**
* Detects AI crawlers by User-Agent (the list is editable in the settings).
* Serves clean Markdown on the same URL with `Content-Type: text/markdown`, `Vary:
User-Agent` and `X-Robots-Tag: noindex`.
* Works with any public post type — including WooCommerce products — plus a Markdown
index for category, tag and custom-taxonomy archives.
* Configurable **YAML front matter** (title, url, date, author, tags, canonical)
with per-field toggles.
* Include your own **custom fields / ACF** values in the front matter.
* Adds machine-readable Schema.org JSON-LD (title, dates, author, cover image,
publisher).
* Generates virtual `/llms.txt` and `/llms-full.txt`.
* Caches the converted Markdown in post meta for fast responses; the cache is cleared
automatically when a post is edited.
* Plays nice with page caches: sets `DONOTCACHEPAGE` (WP Rocket, W3 Total Cache,
WP Super Cache, LiteSpeed) so bots are never served a stale HTML cache.
* Exclude content with a per-post “Hide from AI bots” checkbox or by category/tag
slug.
* “View Markdown” link in the editor and post list — preview exactly what crawlers
receive.
**Privacy and external services**
This plugin does **not** connect to any external service, send any data off your
site, or load any remote scripts. All processing happens on your own server.
**Good to know**
* No .htaccess edits and no Composer — a built-in, zero-dependency HTMLMarkdown
converter is included.
* Does not interfere with SEO plugins, caches, or WooCommerce shop/cart/checkout
pages.
* Private, password-protected and unpublished content is never served to bots —
access rights are not bypassed.
**Pro version**
A Pro version is available with bot IP verification (anti-spoofing), bot-visit analytics
with CSV export, a one-click self-test, and priority support. See the plugin’s settings
page for details. The free version is fully functional on its own and is not time-
limited.
## Screenshots
[⌊The settings screen: content negotiation, editable AI-bot list, and llms.txt options.⌉⌊
The settings screen: content negotiation, editable AI-bot list, and llms.txt options
.⌉[
The settings screen: content negotiation, editable AI-bot list, and llms.txt options.
[⌊The "Hide from AI bots" checkbox on the post editor.⌉⌊The "Hide from AI bots"
checkbox on the post editor.⌉[
The “Hide from AI bots” checkbox on the post editor.
## Installation
1. Upload the `adg-markdown-for-ai-crawlers` folder to `/wp-content/plugins/`, or
install the plugin through the WordPress “Plugins” screen.
2. Activate the plugin.
3. Go to **Settings AI Bots & Markdown** to configure it.
4. On Nginx, if `/llms.txt` returns a 404, make sure your config falls through to `
index.php` (the standard `try_files $uri $uri/ /index.php?$args;` block covers
this). On Apache it works out of the box.
## FAQ
### Will private or draft pages be exposed to bots?
No. Markdown is served only for published, non-password-protected, non-private posts
of the allowed types. Access rights are never bypassed.
### Does it send my content to any external server?
No. There are no outbound requests at all — detection is by User-Agent, and all
conversion happens locally.
### Does it work with page builders like Elementor?
Yes. Content is taken through `the_content`, so shortcodes and builder output are
rendered before conversion. Testing on your specific setup is recommended.
### A bot still receives HTML instead of Markdown. Why?
Most often the request is blocked at the server/CDN level (HTTP 403) or served from
a page cache before the plugin runs. Make sure your cache honors `Vary: User-Agent`
or excludes bot user-agents.
## Reviews
There are no reviews for this plugin.
## Contributors & Developers
“ADG Markdown for AI Crawlers” is open source software. The following people have
contributed to this plugin.
Contributors
* [ andyodua ](https://profiles.wordpress.org/andyodua/)
[Translate “ADG Markdown for AI Crawlers” into your language.](https://translate.wordpress.org/projects/wp-plugins/adg-markdown-for-ai-crawlers)
### Interested in development?
[Browse the code](https://plugins.trac.wordpress.org/browser/adg-markdown-for-ai-crawlers/),
check out the [SVN repository](https://plugins.svn.wordpress.org/adg-markdown-for-ai-crawlers/),
or subscribe to the [development log](https://plugins.trac.wordpress.org/log/adg-markdown-for-ai-crawlers/)
by [RSS](https://plugins.trac.wordpress.org/log/adg-markdown-for-ai-crawlers/?limit=100&mode=stop_on_copy&format=rss).
## Changelog
#### 1.1.0
* New: configurable YAML front matter (title, url, date, author, tags, canonical)
with per-field toggles.
* New: include custom fields / ACF values in the Markdown front matter.
* New: Markdown index for category, tag and custom-taxonomy archives.
* New: WooCommerce products (and any public post type) can be served as Markdown
and listed in llms.txt.
* New: exclude content by category/tag slug, in addition to the per-post checkbox.
* New: “View Markdown” preview link in the post editor and post list.
* Improved: page-cache compatibility — sets DONOTCACHEPAGE for WP Rocket, W3 Total
Cache, WP Super Cache and LiteSpeed.
* Fixed: list item text (simple entries) was dropped from the Markdown output.
#### 1.0.0
* Initial release: content negotiation (Markdown for AI bots on the same URL),
Schema.org/JSON-LD front matter, post-level Markdown cache, virtual llms.txt /
llms-full.txt, and a “Hide from AI bots” metabox.
## Meta
* Version **1.1.0**
* Last updated **2 days ago**
* Active installations **Fewer than 10**
* WordPress version ** 6.0 or higher **
* Tested up to **7.1**
* PHP version ** 7.4 or higher **
* Tags
* [AI](https://wordpress.org/plugins/tags/ai/)[crawler](https://wordpress.org/plugins/tags/crawler/)
[gptbot](https://wordpress.org/plugins/tags/gptbot/)[llms.txt](https://wordpress.org/plugins/tags/llms-txt/)
[markdown](https://wordpress.org/plugins/tags/markdown/)
* [Advanced View](https://wordpress.org/plugins/adg-markdown-for-ai-crawlers/advanced/)
## Ratings
No reviews have been submitted yet.
[Your review](https://wordpress.org/support/plugin/adg-markdown-for-ai-crawlers/reviews/#new-post)
[See all reviews](https://wordpress.org/support/plugin/adg-markdown-for-ai-crawlers/reviews/)
## Contributors
* [ andyodua ](https://profiles.wordpress.org/andyodua/)
## Support
Got something to say? Need help?
[View support forum](https://wordpress.org/support/plugin/adg-markdown-for-ai-crawlers/)