CrawlBrief for LLMs.txt

Description

CrawlBrief for LLMs.txt helps site owners curate an AI-readable content map and publish it as llms.txt at the site root.

Instead of treating llms.txt as an unreviewed URL dump, the plugin combines automatic discovery with editorial control. Administrators can define high-priority Start Here links, mark individual resources as featured or excluded, write concise descriptions, suppress selected category sections, choose public post types and add optional trust or policy links.

Optional virtual Markdown mirrors expose clean representations of public posts, pages and other configured public post types at index.html.md endpoints. The plugin serves both formats through WordPress rewrite rules, so it does not create or overwrite physical files.

The plugin does not send site content to an external API and does not add public footer credits or backlinks.

Features

• Publishes /llms.txt automatically.
• Optional Markdown endpoints, such as /example-post/index.html.md.
• Focused settings page under Settings > CrawlBrief for LLMs.txt.
• Configurable site title, summary and detailed context.
• Configurable post types.
• Configurable link limit for automatically generated sections.
• Configurable excluded category section slugs.
• Editable Start Here and Optional link sections.
• Per-resource Automatic, Featured and Excluded editorial states.
• Dedicated Featured Resources section for manually prioritized content.
• Optional generated-output caching.
• Daily regeneration option.
• Cache purge integration with common WordPress caching plugins.
• Custom per-post editorial status and llms.txt description meta box.
• Uninstall cleanup for plugin options and generated cache.

LLMs.txt Lab Tools

CrawlBrief generates and manages llms.txt inside WordPress. For additional checks or manual workflows, LLMs.txt Lab provides free browser-based tools for working with llms.txt files.

• Validate llms.txt — check an existing llms.txt file or URL.
• Generate llms.txt — manually create an llms.txt file when needed.
• LLMs.txt Lab — access additional llms.txt tools and resources.

These tools are optional. CrawlBrief does not automatically send site content or generated llms.txt data to LLMs.txt Lab.

How to Verify the Plugin

After activation, open Settings > CrawlBrief for LLMs.txt. The settings page shows verification URLs for your website.

To check the generated llms.txt file, visit:

https://example.com/llms.txt

To check the homepage Markdown endpoint, visit:

https://example.com/index.html.md

To check a post or page Markdown endpoint, open any public post or page URL and add index.html.md after the trailing slash:

https://example.com/example-post/index.html.md

The /llms.txt URL should load as plain text. A post or page Markdown URL should load clean Markdown content with the title, canonical URL, last-updated date, author when available, and page body. The homepage Markdown endpoint contains the available homepage or configured site information.

Cache Compatibility

The plugin clears its internal transient cache when configured content changes. It also attempts to purge common caching plugins when available, including LiteSpeed Cache, WP Rocket, W3 Total Cache, WP Super Cache and SiteGround Optimizer.

Other caching plugins can hook into:

crwbrf_txt_cache_purged

Privacy

This plugin does not collect analytics, track visitors or automatically send generated llms.txt content or site data to a third-party service. All generation happens inside WordPress using public site content and settings saved by the site administrator. Links to optional external tools, including LLMs.txt Lab, are opened only when an administrator chooses to visit them. CrawlBrief does not automatically transmit site content or generated llms.txt data through those links.