Description
FocusWeb AI Crawler Monitor is an observation-only plugin that helps you track and understand how AI crawlers interact with your WordPress site, and how much AI referral traffic it sends you. It never blocks or alters requests — it simply logs matching visits and gives you a dashboard to review them. Think of it as AI SEO / answer engine optimization (AEO) visibility for site owners: see which AI crawlers are reading your content, and which AI answer engines (ChatGPT, Perplexity, and more) are actually sending you visitors.
Quick start:
- Install and activate the plugin — no configuration required, logging starts immediately.
- Wait for real traffic (or test with a
curlrequest spoofing a bot User-Agent) — GPTBot, ClaudeBot, and others are detected automatically. - Check AI Crawlers > Dashboard for crawler visits, or AI Crawlers > AI Referrals for human click-throughs from AI answer engines like ChatGPT and Perplexity.
- Generate a suggested llms.txt file from AI Crawlers > llms.txt Generator and place it at your site root.
Detected bots (built in, configurable):
- GPTBot (OpenAI)
- ChatGPT-User (OpenAI, on-demand)
- ClaudeBot (Anthropic)
- Claude-Web (Anthropic)
- PerplexityBot
- Google-Extended
- CCBot (Common Crawl)
- Bytespider (ByteDance)
- Amazonbot
- Applebot-Extended
Features:
- Lightweight
init-hook based detection, no server log file access required. - Custom database table with indexes for fast dashboard queries.
- Dashboard with summary cards, a Chart.js time-series chart, a most-crawled URL table, and a paginated raw log with CSV export.
- Configurable log retention with a daily cleanup cron job.
- Optional IP anonymization (masks the last octet/segment) for GDPR-friendly logging, enabled by default.
- Excludable URL path patterns (e.g.
/wp-admin/*,/checkout/*). - AI Referral Traffic tracking (AEO / answer-engine-optimization visibility): detects and tracks human click-throughs from AI answer engines (ChatGPT, Perplexity, Microsoft Copilot, Google Gemini, Claude, Meta AI, You.com) via the Referer header or matching utm_source parameters, and captures the source query when the referring platform exposes one.
- Built-in llms.txt generator: produces suggested content from your site title, tagline, pages, and posts for manual placement at your site root.
- Fully functional on shared/managed hosting — no filesystem writes outside the database, no special server permissions needed.
Screenshots
Installation
- Upload the
focusweb-ai-crawler-monitorfolder to/wp-content/plugins/. - Activate the plugin through the “Plugins” menu in WordPress.
- On activation, the plugin creates its log table and schedules a daily cleanup cron event.
- Go to AI Crawlers > Dashboard to view logged crawler visits, AI Crawlers > AI Referrals to view human click-throughs from AI answer engines, AI Crawlers > Settings to configure tracked bots/sources, retention, and IP anonymization, and AI Crawlers > llms.txt Generator to produce a suggested llms.txt file.
FAQ
-
Does this plugin block AI crawlers?
-
No. This plugin is observation-only. It never returns a different response, denies a request, or modifies robots.txt. It only records matching visits for your review.
-
Does it need access to my server’s raw access logs?
-
No. Detection happens via a WordPress
inithook reading the User-Agent header on each request, so it works identically on shared and managed hosting. -
Will it log every request from these bots?
-
The plugin skips
wp-admin,wp-login.php,admin-ajax.php, REST API requests, and static asset requests (images, CSS, JS, fonts, etc.) to keep the log focused on actual content visits. You can add further exclusions (e.g./checkout/*) on the Settings page. -
Does it store personal data?
-
It stores the requesting IP address and User-Agent string for each logged visit. IP anonymization (masking the last octet/segment) is enabled by default. You can also configure automatic log deletion after a set number of days.
-
Does the llms.txt generator write files to my server?
-
No. It only generates suggested content in a textarea, with a button to download it as
llms.txt. You place the file at your site root manually, avoiding filesystem permission issues on managed hosting. -
Can I add more bots to track?
-
The bot registry lives in
includes/class-detector.php(AICM_Detector::get_known_bots()) as a simple key/value array, so developers can extend it by adding a new entry with a label and a User-Agent substring to match. -
Can the plugin show me what someone asked ChatGPT/Perplexity that led them to my site?
-
Only when the referring platform happens to expose it, and only for actual human click-throughs — not for crawler visits. AI crawlers (GPTBot, ClaudeBot, etc.) never transmit the triggering question over HTTP, so this is not available for bot traffic. For AI Referral Traffic (a person clicking a citation link in an AI answer), the plugin checks the Referer header for a query parameter; Perplexity is the main platform known to include it. Many platforms, including ChatGPT’s web app, strip the referrer down to bare origin for privacy, so the captured query will often be empty.
Reviews
Contributors & Developers
“FocusWeb AI Crawler Monitor” is open source software. The following people have contributed to this plugin.
Contributors
Translate “FocusWeb AI Crawler Monitor” into your language.
Interested in development?
Browse the code, check out the SVN repository, or subscribe to the development log by RSS.
Changelog
1.2.4
- Directory listing optimization: swapped the oversaturated
seotag foranalytics, added a Quick Start section, and worked in AEO (answer engine optimization) and “track” terminology to better match how site owners search for this plugin on WordPress.org. No functional changes.
1.2.3
- Added plugin directory assets: icon, banner, and screenshots for the WordPress.org listing. No functional changes.
1.2.2
- Plugin Check follow-up: annotated remaining
PluginCheck.Security.DirectDB.UnescapedDBParameterwarnings in class-logger.php with justified ignore comments (all queries use fixed internal table-name identifiers passed through$wpdb->prepare()for any real parameters — no functional change).
1.2.1
- Fixes from WordPress.org plugin review: updated bundled Chart.js from 4.4.4 to the latest stable 4.5.1, and replaced the raw inline script block on the llms.txt Generator page with a properly enqueued
wp_enqueue_script()file (assets/llms-generator.js).
1.2.0
- Added AI Referral Traffic tracking: detects human click-throughs from AI answer engines (ChatGPT, Perplexity, Microsoft Copilot, Google Gemini, Claude, Meta AI, You.com) via the Referer header or matching utm_source parameters, with a best-effort captured query where the referring platform exposes one (most reliably on Perplexity). Adds a new “AI Referrals” dashboard (summary cards, captured-queries table, paginated raw log, CSV export) and per-source tracking toggles on the Settings page. Existing sites will get the new database table automatically on next page load after updating.
1.1.0
- Renamed to “FocusWeb AI Crawler Monitor” and re-slugged to
focusweb-ai-crawler-monitorto comply with WordPress.org’s plugin naming policy (unique/brand-prefixed name required for directory submission). No functional changes.
1.0.1
- Compliance fixes for the WordPress.org Plugin Check tool: removed placeholder Plugin URI/Domain Path headers, dropped the discouraged manual
load_plugin_textdomain()call, justified/annotated direct-DB and nonce-verification warnings, prefixed template-local variables, and updated readme metadata (tags, tested-up-to, short description).
1.0.0
- Initial release: bot detection and logging, admin dashboard with summary cards and Chart.js visualization, most-crawled URL and raw log tables with CSV export, settings page (tracked bots, retention, IP anonymization, excluded paths), and llms.txt generator.