ReadyStack Robots Txt AI Crawler Audit

Description

Checks robots.txt line by line and says what each AI crawler token actually controls – training, AI-search citation, or user fetch – and which lines.

What it does

It reads a robots.txt line by line and reports what each line really controls. It flags tokens that no crawler answers to, such as GPT-Bot or the retired Claude-Web, and separates training crawlers like GPTBot from citation crawlers like OAI-SearchBot, so you can see which lines cost you visibility in AI answers.

How to use it

1. Install and activate the plugin.
2. Edit the page or post where you want the tool and add the shortcode [readystack_robots_txt_ai_crawler_audit]. In the block editor, use a Shortcode block.
3. Open the page, paste the file you want to check into the box, and read the findings listed underneath. Nothing is uploaded.

What you can do with the results

Each finding names the rule it breaks and the fix. Correct the flagged lines, paste the file again, and keep the clean result as a dated record of the check for your checklist, ticket or client report.

Privacy

The tool runs in the browser of whoever opens the page. The plugin makes no external requests, sets no cookies, adds no tracking and stores nothing in your database. It has no settings screen.

Source code

The JavaScript and CSS files in the plugin’s assets folder are the complete, human-readable source. There is no build step: nothing is compiled, bundled or minified, so what you read is what runs.

Free and paid

The free tool on this page is complete for one pasted file at a time: every finding is listed with the rule it breaks and the fix. Nothing in this plugin is locked or time-limited. A separate paid version for teams is sold on our website, not inside this plugin. Full version: https://getreadystack.com/tools/robots-txt-ai-crawler-audit?ref=wporg