{"id":336290,"date":"2026-07-09T09:08:40","date_gmt":"2026-07-09T09:08:40","guid":{"rendered":"https:\/\/wordpress.org\/plugins\/focusweb-ai-crawler-monitor\/"},"modified":"2026-07-10T08:58:13","modified_gmt":"2026-07-10T08:58:13","slug":"focusweb-ai-crawler-monitor","status":"publish","type":"plugin","link":"https:\/\/wordpress.org\/plugins\/focusweb-ai-crawler-monitor\/","author":14225303,"comment_status":"closed","ping_status":"closed","template":"","meta":{"version":"1.4.0","stable_tag":"1.4.0","tested":"7.0.1","requires":"6.0","requires_php":"7.4","requires_plugins":null,"header_name":"FocusWeb AI Crawler Monitor","header_author":"FocusWeb","header_description":"Detects and logs visits from AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, and more) and shows the data in a simple admin dashboard. Also generates a suggested llms.txt file.","assets_banners_color":"133453","last_updated":"2026-07-10 08:58:13","external_support_url":"","external_repository_url":"","donate_link":"","header_plugin_uri":"","header_author_uri":"https:\/\/focusweb.co.il","rating":5,"author_block_rating":0,"active_installs":0,"downloads":74,"num_ratings":1,"support_threads":0,"support_threads_resolved":0,"author_block_count":0,"sections":["description","installation","faq","changelog"],"tags":{"1.2.2":{"tag":"1.2.2","author":"upi","date":"2026-07-09 09:08:21"},"1.2.3":{"tag":"1.2.3","author":"upi","date":"2026-07-09 10:05:31"},"1.2.4":{"tag":"1.2.4","author":"upi","date":"2026-07-09 13:29:10"},"1.4.0":{"tag":"1.4.0","author":"upi","date":"2026-07-10 08:58:13"}},"upgrade_notice":{"1.4.0":"<p>Adds an opt-in auto-update toggle (off by default) and a crawl-to-referral conversion insight with daily chart on the AI Referrals page -- no breaking changes.<\/p>","1.2.4":"<p>Directory listing\/SEO tweaks (tags, description, quick start) -- no functional changes.<\/p>","1.2.3":"<p>Adds plugin directory listing assets (icon, banner, screenshots) -- no functional changes.<\/p>","1.2.2":"<p>Plugin Check annotation follow-up -- no functional changes.<\/p>","1.2.1":"<p>Updates bundled Chart.js to 4.5.1 and fixes inline-script enqueueing per WordPress.org review feedback -- no functional changes.<\/p>","1.2.0":"<p>Adds AI Referral Traffic tracking (new &quot;AI Referrals&quot; page) alongside existing crawler logging -- no breaking changes.<\/p>","1.1.0":"<p>Plugin renamed\/re-slugged for WordPress.org directory compliance -- no functional changes.<\/p>","1.0.1":"<p>Plugin Check compliance fixes -- no functional changes to bot detection or logging.<\/p>"},"ratings":{"1":0,"2":0,"3":0,"4":0,"5":1},"assets_icons":{"icon-128x128.png":{"filename":"icon-128x128.png","revision":3601237,"resolution":"128x128","location":"assets","locale":"","width":128,"height":128},"icon-256x256.png":{"filename":"icon-256x256.png","revision":3601237,"resolution":"256x256","location":"assets","locale":"","width":256,"height":256}},"assets_banners":{"banner-1544x500.png":{"filename":"banner-1544x500.png","revision":3601237,"resolution":"1544x500","location":"assets","locale":"","width":1544,"height":500},"banner-772x250.png":{"filename":"banner-772x250.png","revision":3601237,"resolution":"772x250","location":"assets","locale":"","width":772,"height":250}},"assets_blueprints":{},"all_blocks":[],"tagged_versions":["1.2.2","1.2.3","1.2.4","1.4.0"],"block_files":[],"assets_screenshots":{"screenshot-1.png":{"filename":"screenshot-1.png","revision":3601299,"resolution":"1","location":"assets","locale":"","width":1400,"height":1500},"screenshot-2.png":{"filename":"screenshot-2.png","revision":3601299,"resolution":"2","location":"assets","locale":"","width":1400,"height":955},"screenshot-3.png":{"filename":"screenshot-3.png","revision":3601299,"resolution":"3","location":"assets","locale":"","width":1400,"height":1500},"screenshot-4.png":{"filename":"screenshot-4.png","revision":3601299,"resolution":"4","location":"assets","locale":"","width":1400,"height":900}},"screenshots":{"1":"Dashboard -- summary cards, a 30-day per-bot visit chart, most-crawled URLs, and a paginated raw log with CSV export.","2":"AI Referrals -- human click-throughs from AI answer engines like ChatGPT and Perplexity, a crawl-to-referral conversion insight, a per-day-per-source chart, and captured queries where the platform exposes one.","3":"Settings -- tracked bots, tracked AI referral sources, log retention, IP anonymization, and excluded URL paths.","4":"llms.txt Generator -- produces suggested llms.txt content from your site's pages and posts for manual placement at your site root."}},"plugin_section":[],"plugin_tags":[2353,232,2359,9837,244604],"plugin_category":[55],"plugin_contributors":[270683],"plugin_business_model":[],"class_list":["post-336290","plugin","type-plugin","status-publish","hentry","plugin_tags-ai","plugin_tags-analytics","plugin_tags-bot","plugin_tags-crawler","plugin_tags-llms-txt","plugin_category-seo-and-marketing","plugin_contributors-upi","plugin_committers-upi"],"banners":{"banner":"https:\/\/ps.w.org\/focusweb-ai-crawler-monitor\/assets\/banner-772x250.png?rev=3601237","banner_2x":"https:\/\/ps.w.org\/focusweb-ai-crawler-monitor\/assets\/banner-1544x500.png?rev=3601237","banner_rtl":false,"banner_2x_rtl":false},"icons":{"svg":false,"icon":"https:\/\/ps.w.org\/focusweb-ai-crawler-monitor\/assets\/icon-128x128.png?rev=3601237","icon_2x":"https:\/\/ps.w.org\/focusweb-ai-crawler-monitor\/assets\/icon-256x256.png?rev=3601237","generated":false},"screenshots":[{"src":"https:\/\/ps.w.org\/focusweb-ai-crawler-monitor\/assets\/screenshot-1.png?rev=3601299","caption":"Dashboard -- summary cards, a 30-day per-bot visit chart, most-crawled URLs, and a paginated raw log with CSV export."},{"src":"https:\/\/ps.w.org\/focusweb-ai-crawler-monitor\/assets\/screenshot-2.png?rev=3601299","caption":"AI Referrals -- human click-throughs from AI answer engines like ChatGPT and Perplexity, a crawl-to-referral conversion insight, a per-day-per-source chart, and captured queries where the platform exposes one."},{"src":"https:\/\/ps.w.org\/focusweb-ai-crawler-monitor\/assets\/screenshot-3.png?rev=3601299","caption":"Settings -- tracked bots, tracked AI referral sources, log retention, IP anonymization, and excluded URL paths."},{"src":"https:\/\/ps.w.org\/focusweb-ai-crawler-monitor\/assets\/screenshot-4.png?rev=3601299","caption":"llms.txt Generator -- produces suggested llms.txt content from your site's pages and posts for manual placement at your site root."}],"raw_content":"<!--section=description-->\n<p>FocusWeb AI Crawler Monitor is an observation-only plugin that helps you track and understand how AI crawlers interact with your WordPress site, and how much AI referral traffic it sends you. It never blocks or alters requests -- it simply logs matching visits and gives you a dashboard to review them. Think of it as AI SEO \/ answer engine optimization (AEO) visibility for site owners: see which AI crawlers are reading your content, and which AI answer engines (ChatGPT, Perplexity, and more) are actually sending you visitors.<\/p>\n\n<p><strong>Quick start:<\/strong><\/p>\n\n<ol>\n<li>Install and activate the plugin -- no configuration required, logging starts immediately.<\/li>\n<li>Wait for real traffic (or test with a <code>curl<\/code> request spoofing a bot User-Agent) -- GPTBot, ClaudeBot, and others are detected automatically.<\/li>\n<li>Check <strong>AI Crawlers &gt; Dashboard<\/strong> for crawler visits, or <strong>AI Crawlers &gt; AI Referrals<\/strong> for human click-throughs from AI answer engines like ChatGPT and Perplexity.<\/li>\n<li>Generate a suggested llms.txt file from <strong>AI Crawlers &gt; llms.txt Generator<\/strong> and place it at your site root.<\/li>\n<\/ol>\n\n<p><strong>How to use this data to optimize for AI:<\/strong><\/p>\n\n<ul>\n<li>A high number of crawler visits with few AI referrals is normal and expected today -- most AI crawling feeds training or indexing, not real-time answers. The Dashboard's most-crawled-URL table shows you which pages get the most attention; make sure those pages are accurate, current, and well-structured.<\/li>\n<li>The AI Referrals page's Crawl-to-Referral Gap panel puts a number on that relationship, so you can track whether it improves over time.<\/li>\n<li>Check the Captured Queries table for the (best-effort) questions that led people to your site -- these are real signals of what your audience is asking AI assistants, and can guide what to write about next.<\/li>\n<li>Favor clear headings, concise direct answers, and FAQ-style content -- AI answer engines tend to extract and cite well-structured, unambiguous text more readily than dense marketing copy.<\/li>\n<li>Use the llms.txt Generator to publish a curated, machine-readable summary of your key pages -- see the FAQ below on llms.txt's actual adoption status before treating it as a guaranteed lever.<\/li>\n<\/ul>\n\n<p><strong>Detected bots (built in, configurable):<\/strong><\/p>\n\n<ul>\n<li>GPTBot (OpenAI)<\/li>\n<li>ChatGPT-User (OpenAI, on-demand)<\/li>\n<li>ClaudeBot (Anthropic)<\/li>\n<li>Claude-Web (Anthropic)<\/li>\n<li>PerplexityBot<\/li>\n<li>Google-Extended<\/li>\n<li>CCBot (Common Crawl)<\/li>\n<li>Bytespider (ByteDance)<\/li>\n<li>Amazonbot<\/li>\n<li>Applebot-Extended<\/li>\n<\/ul>\n\n<p><strong>Features:<\/strong><\/p>\n\n<ul>\n<li>Lightweight <code>init<\/code>-hook based detection, no server log file access required.<\/li>\n<li>Custom database table with indexes for fast dashboard queries.<\/li>\n<li>Dashboard with summary cards, a Chart.js time-series chart, a most-crawled URL table, and a paginated raw log with CSV export.<\/li>\n<li>Configurable log retention with a daily cleanup cron job.<\/li>\n<li>Optional IP anonymization (masks the last octet\/segment) for GDPR-friendly logging, enabled by default.<\/li>\n<li>Excludable URL path patterns (e.g. <code>\/wp-admin\/*<\/code>, <code>\/checkout\/*<\/code>).<\/li>\n<li>AI Referral Traffic tracking (AEO \/ answer-engine-optimization visibility): detects and tracks human click-throughs from AI answer engines (ChatGPT, Perplexity, Microsoft Copilot, Google Gemini, Claude, Meta AI, You.com) via the Referer header or matching utm_source parameters, captures the source query when the referring platform exposes one, and shows a crawl-to-referral conversion insight alongside a per-day-per-source chart.<\/li>\n<li>Built-in llms.txt generator: produces suggested content from your site title, tagline, pages, and posts for manual placement at your site root (see FAQ for llms.txt's actual adoption status).<\/li>\n<li>Opt-in auto-update toggle right on the Settings page -- stays in sync with WordPress core's native \"Enable auto-updates\" link on the Plugins screen.<\/li>\n<li>Fully functional on shared\/managed hosting -- no filesystem writes outside the database, no special server permissions needed.<\/li>\n<\/ul>\n\n<p><strong>Roadmap (planned direction, not a commitment to specific versions or dates):<\/strong><\/p>\n\n<ul>\n<li>Correlating individual crawler visits with later referral visits to the same page, to see which crawled content actually converts into AI-driven traffic.<\/li>\n<li>Per-post\/per-page llms.txt summaries, instead of one sitewide file.<\/li>\n<li>Additional AI referral sources as new answer engines emerge.<\/li>\n<li>A combined \"AI visibility\" trend view comparing crawl volume and referral traffic side by side over longer time ranges.<\/li>\n<li>Revisiting llms.txt's role in this plugin if\/when major AI platforms confirm they actually consume it.<\/li>\n<\/ul>\n\n<p>Suggestions are welcome via the support forum.<\/p>\n\n<!--section=installation-->\n<ol>\n<li>Upload the <code>focusweb-ai-crawler-monitor<\/code> folder to <code>\/wp-content\/plugins\/<\/code>.<\/li>\n<li>Activate the plugin through the \"Plugins\" menu in WordPress.<\/li>\n<li>On activation, the plugin creates its log table and schedules a daily cleanup cron event.<\/li>\n<li>Go to <strong>AI Crawlers &gt; Dashboard<\/strong> to view logged crawler visits, <strong>AI Crawlers &gt; AI Referrals<\/strong> to view human click-throughs from AI answer engines, <strong>AI Crawlers &gt; Settings<\/strong> to configure tracked bots\/sources, retention, and IP anonymization, and <strong>AI Crawlers &gt; llms.txt Generator<\/strong> to produce a suggested llms.txt file.<\/li>\n<\/ol>\n\n<!--section=faq-->\n<dl>\n<dt id=\"does%20this%20plugin%20block%20ai%20crawlers%3F\"><h3>Does this plugin block AI crawlers?<\/h3><\/dt>\n<dd><p>No. This plugin is observation-only. It never returns a different response, denies a request, or modifies robots.txt. It only records matching visits for your review.<\/p><\/dd>\n<dt id=\"does%20it%20need%20access%20to%20my%20server%27s%20raw%20access%20logs%3F\"><h3>Does it need access to my server's raw access logs?<\/h3><\/dt>\n<dd><p>No. Detection happens via a WordPress <code>init<\/code> hook reading the User-Agent header on each request, so it works identically on shared and managed hosting.<\/p><\/dd>\n<dt id=\"will%20it%20log%20every%20request%20from%20these%20bots%3F\"><h3>Will it log every request from these bots?<\/h3><\/dt>\n<dd><p>The plugin skips <code>wp-admin<\/code>, <code>wp-login.php<\/code>, <code>admin-ajax.php<\/code>, REST API requests, and static asset requests (images, CSS, JS, fonts, etc.) to keep the log focused on actual content visits. You can add further exclusions (e.g. <code>\/checkout\/*<\/code>) on the Settings page.<\/p><\/dd>\n<dt id=\"does%20it%20store%20personal%20data%3F\"><h3>Does it store personal data?<\/h3><\/dt>\n<dd><p>It stores the requesting IP address and User-Agent string for each logged visit. IP anonymization (masking the last octet\/segment) is enabled by default. You can also configure automatic log deletion after a set number of days.<\/p><\/dd>\n<dt id=\"does%20the%20llms.txt%20generator%20write%20files%20to%20my%20server%3F\"><h3>Does the llms.txt generator write files to my server?<\/h3><\/dt>\n<dd><p>No. It only generates suggested content in a textarea, with a button to download it as <code>llms.txt<\/code>. You place the file at your site root manually, avoiding filesystem permission issues on managed hosting.<\/p><\/dd>\n<dt id=\"can%20i%20add%20more%20bots%20to%20track%3F\"><h3>Can I add more bots to track?<\/h3><\/dt>\n<dd><p>The bot registry lives in <code>includes\/class-detector.php<\/code> (<code>AICM_Detector::get_known_bots()<\/code>) as a simple key\/value array, so developers can extend it by adding a new entry with a label and a User-Agent substring to match.<\/p><\/dd>\n<dt id=\"can%20the%20plugin%20show%20me%20what%20someone%20asked%20chatgpt%2Fperplexity%20that%20led%20them%20to%20my%20site%3F\"><h3>Can the plugin show me what someone asked ChatGPT\/Perplexity that led them to my site?<\/h3><\/dt>\n<dd><p>Only when the referring platform happens to expose it, and only for actual human click-throughs -- not for crawler visits. AI crawlers (GPTBot, ClaudeBot, etc.) never transmit the triggering question over HTTP, so this is not available for bot traffic. For AI Referral Traffic (a person clicking a citation link in an AI answer), the plugin checks the Referer header for a query parameter; Perplexity is the main platform known to include it. Many platforms, including ChatGPT's web app, strip the referrer down to bare origin for privacy, so the captured query will often be empty.<\/p><\/dd>\n<dt id=\"is%20llms.txt%20an%20officially%20supported%20standard%3F\"><h3>Is llms.txt an officially supported standard?<\/h3><\/dt>\n<dd><p>Not yet. llms.txt is a community-proposed convention (introduced in 2024) suggesting a simple Markdown file at your site root to help AI systems find your key content. As of today, no major AI crawler or assistant (OpenAI, Anthropic, Google, Perplexity, etc.) has publicly confirmed that it reads or prioritizes llms.txt in production. Publishing one is a reasonable, low-effort bet on where things may be heading -- not a guaranteed way to influence how AI systems treat your site. This plugin only generates suggested content for you to review and place manually; it never assumes llms.txt is authoritative or required.<\/p><\/dd>\n\n<\/dl>\n\n<!--section=changelog-->\n<h4>1.4.0<\/h4>\n\n<ul>\n<li>Added an opt-in auto-update toggle to the Settings page. It reads and writes WordPress core's own <code>auto_update_plugins<\/code> list directly, so it always stays in sync with the native \"Enable auto-updates\" link on the Plugins screen -- off by default, same as core.<\/li>\n<li>Added a \"Crawl-to-Referral Gap\" insight and a per-day-per-source chart to the AI Referrals page, showing how much AI crawling actually converts into real referral traffic.<\/li>\n<li>Expanded the readme with guidance on acting on this data, an honest note on llms.txt's actual (unconfirmed) adoption status by AI platforms, and a roadmap section.<\/li>\n<\/ul>\n\n<h4>1.2.4<\/h4>\n\n<ul>\n<li>Directory listing optimization: swapped the oversaturated <code>seo<\/code> tag for <code>analytics<\/code>, added a Quick Start section, and worked in AEO (answer engine optimization) and \"track\" terminology to better match how site owners search for this plugin on WordPress.org. No functional changes.<\/li>\n<\/ul>\n\n<h4>1.2.3<\/h4>\n\n<ul>\n<li>Added plugin directory assets: icon, banner, and screenshots for the WordPress.org listing. No functional changes.<\/li>\n<\/ul>\n\n<h4>1.2.2<\/h4>\n\n<ul>\n<li>Plugin Check follow-up: annotated remaining <code>PluginCheck.Security.DirectDB.UnescapedDBParameter<\/code> warnings in class-logger.php with justified ignore comments (all queries use fixed internal table-name identifiers passed through <code>$wpdb-&gt;prepare()<\/code> for any real parameters -- no functional change).<\/li>\n<\/ul>\n\n<h4>1.2.1<\/h4>\n\n<ul>\n<li>Fixes from WordPress.org plugin review: updated bundled Chart.js from 4.4.4 to the latest stable 4.5.1, and replaced the raw inline script block on the llms.txt Generator page with a properly enqueued <code>wp_enqueue_script()<\/code> file (assets\/llms-generator.js).<\/li>\n<\/ul>\n\n<h4>1.2.0<\/h4>\n\n<ul>\n<li>Added AI Referral Traffic tracking: detects human click-throughs from AI answer engines (ChatGPT, Perplexity, Microsoft Copilot, Google Gemini, Claude, Meta AI, You.com) via the Referer header or matching utm_source parameters, with a best-effort captured query where the referring platform exposes one (most reliably on Perplexity). Adds a new \"AI Referrals\" dashboard (summary cards, captured-queries table, paginated raw log, CSV export) and per-source tracking toggles on the Settings page. Existing sites will get the new database table automatically on next page load after updating.<\/li>\n<\/ul>\n\n<h4>1.1.0<\/h4>\n\n<ul>\n<li>Renamed to \"FocusWeb AI Crawler Monitor\" and re-slugged to <code>focusweb-ai-crawler-monitor<\/code> to comply with WordPress.org's plugin naming policy (unique\/brand-prefixed name required for directory submission). No functional changes.<\/li>\n<\/ul>\n\n<h4>1.0.1<\/h4>\n\n<ul>\n<li>Compliance fixes for the WordPress.org Plugin Check tool: removed placeholder Plugin URI\/Domain Path headers, dropped the discouraged manual <code>load_plugin_textdomain()<\/code> call, justified\/annotated direct-DB and nonce-verification warnings, prefixed template-local variables, and updated readme metadata (tags, tested-up-to, short description).<\/li>\n<\/ul>\n\n<h4>1.0.0<\/h4>\n\n<ul>\n<li>Initial release: bot detection and logging, admin dashboard with summary cards and Chart.js visualization, most-crawled URL and raw log tables with CSV export, settings page (tracked bots, retention, IP anonymization, excluded paths), and llms.txt generator.<\/li>\n<\/ul>","raw_excerpt":"Detect, log, and track AI crawler visits (GPTBot, ClaudeBot, PerplexityBot, and more) with a simple admin dashboard.","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/wordpress.org\/plugins\/wp-json\/wp\/v2\/plugin\/336290","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wordpress.org\/plugins\/wp-json\/wp\/v2\/plugin"}],"about":[{"href":"https:\/\/wordpress.org\/plugins\/wp-json\/wp\/v2\/types\/plugin"}],"replies":[{"embeddable":true,"href":"https:\/\/wordpress.org\/plugins\/wp-json\/wp\/v2\/comments?post=336290"}],"author":[{"embeddable":true,"href":"https:\/\/wordpress.org\/plugins\/wp-json\/wporg\/v1\/users\/upi"}],"wp:attachment":[{"href":"https:\/\/wordpress.org\/plugins\/wp-json\/wp\/v2\/media?parent=336290"}],"wp:term":[{"taxonomy":"plugin_section","embeddable":true,"href":"https:\/\/wordpress.org\/plugins\/wp-json\/wp\/v2\/plugin_section?post=336290"},{"taxonomy":"plugin_tags","embeddable":true,"href":"https:\/\/wordpress.org\/plugins\/wp-json\/wp\/v2\/plugin_tags?post=336290"},{"taxonomy":"plugin_category","embeddable":true,"href":"https:\/\/wordpress.org\/plugins\/wp-json\/wp\/v2\/plugin_category?post=336290"},{"taxonomy":"plugin_contributors","embeddable":true,"href":"https:\/\/wordpress.org\/plugins\/wp-json\/wp\/v2\/plugin_contributors?post=336290"},{"taxonomy":"plugin_business_model","embeddable":true,"href":"https:\/\/wordpress.org\/plugins\/wp-json\/wp\/v2\/plugin_business_model?post=336290"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}