Description
Softglaze AI Crawler Monitor records every request from a known AI crawler that reaches WordPress. It shows you which crawlers came, what they asked for, and what your site told them at the time.
Most plugins in this area stop at writing a rule. This one is built around the gap between writing a rule and the rule working, because on the open web those are not the same thing.
You can find out:
- Which AI crawlers requested pages on your site, how often, and which pages.
- What each crawler was told, and whether it took any notice.
- Which requests were refused, if you chose to refuse them.
What it can and cannot do
There are three ways to tell a crawler to stay away. Only one of them is enforced, and the plugin repeats that on every screen where you make the choice.
- A robots.txt rule. This is a request, not a barrier. Well behaved crawlers honour it. One that ignores it can still fetch your pages, and nothing on your server stops it.
- An
X-Robots-Tag: noai, noimageaiheader, sent only on requests from that crawler. Also a request, and a weaker one, becausenoaiis not an agreed standard and most crawlers ignore it. - Refusing the request with a 403. This is the only setting that is actually enforced. A matching request gets an empty “403 Forbidden” and the page is never sent.
The header never contains noindex, nofollow or none under any setting. Those would take your pages out of ordinary Google and Bing results, which is not what anyone means when they say they want to block AI.
Three crawlers in the list are documented by their own operators as fetching pages when a person asks a question, with robots.txt rules that may not apply to them: ChatGPT-User, Perplexity-User and Meta-ExternalFetcher. All three are marked on the crawler screen, right next to the setting, so you find out before you rely on it rather than afterwards.
Your robots.txt is always served, even to a crawler you are refusing. Blocking a crawler from reading robots.txt would leave it no way to learn that you want it to stay away.
Per-content control
The site-wide setting is the default. Any single post, page or product can override it from the edit screen, from quick edit, or from bulk edit. A column in the posts list shows which content has its own setting, and you can filter by it.
A per-post setting sends the X-Robots-Tag header and does nothing else. It adds no lines to robots.txt, it cannot refuse a request, and it cannot lift a refusal you set site-wide. The edit screen says so in those words.
What it cannot see
The plugin records requests that reach WordPress. Plenty of requests never get that far, and rather than quietly under-report, it tells you so:
- Requests your CDN or firewall answers before they reach your server.
- Requests served from a page cache. On a test server with page caching on, only one request in five reached WordPress at all.
- Static files. Images, stylesheets, scripts and fonts are served without running WordPress.
- Addresses on your domain that WordPress does not handle.
- Any crawler that is not in this plugin’s list.
There is a banner above every figure in the plugin saying this. You cannot dismiss it, because it changes what the numbers mean.
Crawler identity checking
A user agent is self declared, so anything can claim to be GPTBot. The plugin can check whether a crawler really is what it says.
Both checks are switched off until you turn them on, because both mean contacting a server outside your site. Left off, the plugin still does everything else, and crawlers are recorded as “not checked”, which is what they honestly are.
A crawler is only ever recorded as having failed a check when a check actually ran and gave a negative answer. A check that could not run is never treated as a failure.
Checking only works where the operator publishes something to check against. Eleven of the crawlers in the list can be checked. The rest stay “not checked” however you set these options, because their operators publish nothing to check them with, and the crawler screen tells you which is which.
Crawlers included in this version
GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, PerplexityBot, Perplexity-User, meta-externalagent, Meta-WebIndexer, Meta-ExternalFetcher, Applebot, Applebot-Extended, Bytespider, CCBot, Amazonbot, Amzn-SearchBot, Amzn-User, cohere-ai, Diffbot, Diffbot-User, Timpibot, YouBot.
Every one of these was checked against the operator’s own published documentation, and each entry records the page it came from. The crawler screen links to that page for each one.
Three of them have no published documentation at all: Bytespider, cohere-ai and Timpibot. They are in the list because the requests are real and you are entitled to see them, and each is labelled on screen as undocumented, since nobody has stated whether they read robots.txt.
Crawler names and robots.txt tokens change over time. The list ships with the plugin, it is versioned, and it is never fetched from a remote server.
Other plugins by SoftGlaze
- SoftGlaze PDF Invoices for WooCommerce — invoices, packing slips, proformas and credit notes from your WooCommerce orders, rendered locally.
- SoftGlaze Maintenance Mode & Coming Soon — coming soon and maintenance pages with templates, countdowns and access control.
- SoftGlaze Click to Chat — a WhatsApp chat button with multiple agents, schedules and WooCommerce support.
External services
On a default install this plugin contacts nothing at all.
It has two features that make an outbound request. Both are switched off when the plugin is installed, and neither one does anything until you tick its box on the settings screen. Neither runs while a page is being served; both run only on a scheduled task in the background.
There is no analytics, no telemetry, no licence check and no update check. Nothing is ever sent to Softglaze or to the plugin’s author, under any setting.
1. Reverse address lookups (DNS) — off by default
What it is and what it is for: a reverse DNS lookup, used to confirm that a crawler’s address really belongs to the company its user agent claims. A user agent is self declared, so anything can claim to be a given crawler; this is how that claim is tested.
Where it goes: this does not contact a service the plugin chooses. It asks the domain name resolver your server is already configured to use, which will be your host’s or your network’s. Because the plugin does not select the provider, there is no third party service here with terms or a privacy policy for us to link to. Which resolver your server uses, and that operator’s terms, are a matter between you and your host.
What data is sent, and when: the crawler’s IP address, and nothing else. This is the only feature in the plugin that sends any visitor information anywhere. It is sent on a scheduled task that runs at most every fifteen minutes, and only for addresses belonging to crawler requests already recorded on your site. The address of an ordinary human visitor is never looked up, because an ordinary visitor is never recorded in the first place.
Under which conditions: only when you have switched “Check crawler identity by reverse address lookup” on. With it off, crawlers are recorded as “not checked” and no address goes anywhere.
Which crawlers this affects: one, YouBot, because you.com is the only operator in the current list that publishes a hostname pattern to check an address against. For every other crawler in the list this setting does nothing at all.
2. Published address lists (HTTPS) — off by default
What it is and what it is for: some AI companies publish, on their own website, the list of IP addresses their crawlers use. When you switch this on, the plugin downloads those files and checks recorded crawler addresses against them. This is the second, independent way of testing a self declared user agent.
What data is sent, and when: nothing about your site, nothing about you, and nothing about any visitor. It is an ordinary unauthenticated GET request for a public file — the same request your own browser would make if you typed the address into it. No identifier, no site URL and no query string is attached. Each file is downloaded at most once per operator per day and the result is stored on your server.
Under which conditions: only when you have switched “Check crawler identity against published address lists” on.
These, in full, are the only addresses this setting ever requests. There are no others, and the plugin never contacts an address that is not in this list.
OpenAI, for GPTBot, OAI-SearchBot and ChatGPT-User.
Files requested: https://openai.com/gptbot.json , https://openai.com/searchbot.json , https://openai.com/chatgpt-user.json
Terms of use: https://openai.com/policies/terms-of-use/
Privacy policy: https://openai.com/policies/privacy-policy/
Anthropic, for ClaudeBot, Claude-User and Claude-SearchBot.
File requested: https://claude.com/crawling/bots.json
Terms of use: https://www.anthropic.com/legal/consumer-terms
Privacy policy: https://www.anthropic.com/legal/privacy
Apple, for Applebot.
File requested: https://search.developer.apple.com/applebot.json
Terms of use: https://www.apple.com/legal/internet-services/
Privacy policy: https://www.apple.com/legal/privacy/
Perplexity, for PerplexityBot and Perplexity-User.
Files requested: https://www.perplexity.ai/perplexitybot.json , https://www.perplexity.ai/perplexity-user.json
Terms of use: https://www.perplexity.ai/hub/legal/terms-of-service
Privacy policy: https://www.perplexity.ai/hub/legal/privacy-notice
Common Crawl, for CCBot.
File requested: https://index.commoncrawl.org/ccbot.json
Terms of use: https://commoncrawl.org/terms-of-use
Privacy policy: https://commoncrawl.org/privacy-policy
Documentation links, which are not requests
Separately from the two features above, the Crawler access screen shows a “documentation” link beside each crawler, pointing at the page on the operator’s own website where that crawler’s behaviour is documented. These are ordinary links in the page. The plugin never fetches them, on any setting, and nothing is requested unless you choose to click one and open it in your own browser.
They are listed here so that the presence of these addresses in the plugin’s source is not mistaken for an undisclosed service. The pages linked are Amazon’s Amazonbot page, Anthropic’s crawler support article, Apple’s Applebot page, Common Crawl’s CCBot page, Diffbot’s robots.txt FAQ, Google’s crawlers page, Meta’s web crawlers page, OpenAI’s bots page, Perplexity’s bots page and you.com’s YouBot page. Three crawlers — Bytespider, cohere-ai and Timpibot — have no documentation link, because their operators publish none.
Every address in this section is also recorded in the plugin folder at includes/data/bot-registry.php, where each crawler carries the documentation page it was checked against and the address list published for it, if there is one. The registry ships with the plugin and is never fetched from a remote server.
Privacy
Visitor IP addresses are never stored in the request log. Each record holds a one way hash instead, made with a random value that is generated when the plugin is installed on your site and is never shared. The hash cannot be turned back into an address.
The raw user agent is not stored either. It is matched against the crawler list, recorded as a crawler name, and then discarded.
There is one exception, and it only exists if you switch on reverse address lookups:
- A crawler’s full address is held in a separate table for as long as the lookup takes, then deleted straight away.
- Anything left over is removed after 60 minutes whether it was checked or not, including on sites where scheduled tasks are not running.
- Deactivating the plugin empties that table immediately.
- It never appears on any screen or in any export.
With reverse address lookups off, which is the default, that table is never written to at all.
How long things are kept is not the same for everything, and the settings screen spells this out:
- Individual request records: 90 days by default, adjustable between 7 and 365.
- Daily totals per crawler: kept indefinitely. Counts only, with no visitor information.
- Daily list of most requested addresses: 400 days. Counts and addresses only.
- Identity check results: 7 days.
- The short lived lookup queue described above: 60 minutes at most.
There is no analytics and no telemetry. Nothing about your site or your visitors is ever sent to the plugin’s author.
Screenshots
Installation
- Install through Plugins, Add New, Upload Plugin, or copy the folder to
/wp-content/plugins/. - Activate it on the Plugins screen.
- Open “AI Crawlers” in the admin menu.
Nothing is blocked and nothing is disallowed when you activate it. The plugin starts by recording what happens and changes nothing about your site until you pick a setting, so you can look before you decide anything.
On activation it writes a small helper file into wp-content/mu-plugins/ so it can identify a crawler at the earliest point WordPress allows. It removes that file again when you deactivate the plugin. If the folder cannot be written to, the plugin detects slightly later in the page load instead and tells you which mode is running. Nothing is lost either way.
FAQ
-
Does this stop AI companies using my content?
-
No, and no plugin can honestly promise that.
Two of the three mechanisms are requests a crawler is free to ignore. The third refuses matching requests at your server and is enforced, but it matches on the user agent the request claims, and a user agent can be faked. A scraper that does not announce itself is unaffected. This is not a firewall.
What you get instead is evidence. If a crawler kept coming after you told it not to, you will see it.
-
Why do my numbers look low?
-
Almost certainly page caching. When a page comes from a cache, WordPress never runs, so the request is not recorded. The plugin has no way to know how much it is missing, so it says that plainly instead of guessing.
-
Why does the top pages list say it is approximate?
-
Because it is. Daily totals keep the 100 most requested addresses per crawler per day, so a page that never reaches a daily top 100 stays invisible in a longer list no matter how many requests it collects. Every row links to the URL checker, which counts individual records and is exact.
-
Can I submit my site to ChatGPT or Claude?
-
No. Neither company offers a submission form, an endpoint or an API, and any plugin that says otherwise is describing something that does not exist.
What OpenAI documents is that any public website can appear in ChatGPT search, and that the way to be eligible is simply not to block their search crawler. Claude’s web search runs on a third party search index rather than one Anthropic takes submissions to.
So the only lever you have is the one this plugin already gives you, which is whether a crawler is allowed. That is why the crawler screen warns you before you disallow a crawler that feeds a product which links back to its sources.
-
What happens to my data if I delete the plugin?
-
Nothing is deleted unless you tick the box on the settings screen first. Deactivating the plugin never deletes anything.
-
Does it work on multisite?
-
Per site. Activate it on each site individually. Network activation is refused with a message explaining why, because everything the plugin stores belongs to one site.
Reviews
There are no reviews for this plugin.
Contributors & Developers
“Softglaze AI Crawler Monitor” is open source software. The following people have contributed to this plugin.
Contributors
Translate “Softglaze AI Crawler Monitor” into your language.
Interested in development?
Browse the code, check out the SVN repository, or subscribe to the development log by RSS.
Changelog
0.7.0
- Renamed to Softglaze AI Crawler Monitor. The previous name described the plugin rather than identifying it, so it now leads with the Softglaze name. Every identifier moved with it, including the database tables, so this version installs as a new plugin rather than upgrading the old one.
- The helper file in
wp-content/mu-plugins/no longer assumes the plugin folder’s name when locating the plugin. It uses the real path recorded when the file was written, and the plugin rewrites that path by itself if the folder is ever moved or renamed. - The “External services” section of this readme now states, for each outbound request, exactly what is sent, when, and under what condition, and separates the operator documentation links shown on screen — which the plugin never fetches — from the addresses it actually requests.
0.6.0
- Per-content control. Set AI access on a single post, page or product from the edit screen, quick edit or bulk edit, with a filterable column in the posts list.
- The URL checker now reports conflicts, meaning which crawlers requested an address after being asked not to, and which were refused.
- The URL checker also shows what each crawler is currently told about that one address, and whether the address is reachable at all.
- Six more crawlers, each checked against its operator’s documentation: Meta-WebIndexer, Meta-ExternalFetcher, Applebot, Amzn-SearchBot, Amzn-User and Diffbot-User.
- The crawler screen now warns you before you disallow a crawler that feeds a product which links back to its sources, because there is no way to submit your site to those products afterwards.
0.5.0
- Every crawler in the list checked against its operator’s own documentation, with the source page recorded and linked from the crawler screen.
- Crawlers whose operators say robots.txt may not apply to them are now marked as such, next to the setting.
- Crawlers whose operators publish no documentation at all are now marked as such.
- Identity checking now works wherever an operator publishes a way to check, covering ten crawlers by published address list or reverse DNS.
- FacebookBot removed, because Meta’s crawler documentation no longer lists it.
- Applebot-Extended corrected to a robots.txt name only, which is what Apple documents it as. It could never have appeared in reports, and it no longer looks as though it might.
0.4.0
- Violation reporting, with separate panels for crawlers that ignored your robots.txt and requests that were refused.
- Crawler identity checking by reverse address lookup and published address lists. Both switched off by default.
- Optional removal of all data when the plugin is deleted. Off by default.
0.3.0
- Choose what each crawler is told: allow, monitor, or disallow.
- robots.txt rules and the
X-Robots-Tagheader for disallowed crawlers. - Optional refusal of matching requests with a 403.
- Detection and handling of an existing robots.txt file at the site root.
0.2.0
- Dashboard showing requests per crawler over 7, 30 and 90 days, with a trend and the most requested addresses.
- URL checker for exact per-address counts.
- Cache state banner.
0.1.0
- Crawler detection and request recording.
- Daily totals and retention cleanup.