Description
AI answer engines can only cite pages they can read. This plugin checks, from inside your dashboard, whether they can read yours.
Go to Tools › AI Crawler Access and press Run the check. It looks at:
- robots.txt for search and answer bots. Googlebot, Bingbot, OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User. Rules are matched the way RFC 9309 says (the longest rule wins, Allow wins a tie), against your home page, newest post and newest page.
- A live request as each bot. robots.txt can allow a bot while a firewall, security plugin or CDN still refuses it. The plugin requests your home page with each bot’s user agent and compares the answer with a normal browser request. It also spots bot challenge pages.
- Indexing and snippet controls.
noindex,nosnippetandmax-snippet:0in meta tags and the X-Robots-Tag header, a canonical that points to another site, and the “Discourage search engines” setting. Google says a page must be indexed and eligible for a snippet to appear in AI Overviews and AI Mode. - XML sitemap, and whether robots.txt lists it.
- Text in the raw HTML. Most AI crawlers do not run JavaScript, so text added by scripts is invisible to them.
- HTTPS, and whether http:// redirects to https://.
Two things are reported but never scored:
- AI training bots (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, meta-externalagent). Blocking these is your choice. It does not remove you from search results or AI answers, so it does not count against you.
- llms.txt. Google has said its Search features do not use it, and no major AI engine has said it reads the file when choosing what to cite. The plugin tells you whether you have one and leaves it at that.
What it cannot tell you
Whether any engine actually cites you. Being reachable is the first step, not the result. The plugin also runs from your own server, so blocks based on the bots’ IP addresses will not show. A clean live test means no user-agent rule is blocking them.
Privacy
Every request the plugin makes goes to your own site. It sends nothing to any other server, sets no cookies on visitors and adds nothing to the front end. The last result is saved in your user profile and deleted when you uninstall the plugin.
Installation
- Install from Plugins › Add New, or upload the folder to
/wp-content/plugins/. - Activate it.
- Go to Tools › AI Crawler Access and press Run the check.
FAQ
-
Why is Google-Extended not marked as a problem?
-
Google-Extended only controls whether Google may use your content to train Gemini models. It does not affect Google Search, AI Overviews or AI Mode. Those use Googlebot.
-
A bot shows “Refused” but my robots.txt allows it. Why?
-
Something in front of WordPress is blocking that user agent: a security plugin, a firewall rule at your host, or a CDN bot setting. Check those for rules that name the bot, or for a general “block AI bots” switch.
-
Everything passes. Why am I still not cited?
-
Reachable is necessary, not enough. Which pages AI answers cite depends mostly on what other sites say about you and whether your pages answer the question better than others. The method page linked in the plugin has the data behind this.
-
Does it slow my site down?
-
No. Nothing runs on the front end. The check runs only when an administrator presses the button.
Reviews
There are no reviews for this plugin.
Contributors & Developers
“Softrecs AI Crawler Access Check” is open source software. The following people have contributed to this plugin.
Contributors
Translate “Softrecs AI Crawler Access Check” into your language.
Interested in development?
Browse the code, check out the SVN repository, or subscribe to the development log by RSS.
Changelog
1.0.0
- First release.