{"id":391148,"date":"2026-10-26T07:48:22","date_gmt":"2026-10-26T07:48:22","guid":{"rendered":"https:\/\/wordpress.org\/plugins\/hitmind-ai-crawler-check\/"},"modified":"2026-10-26T07:48:08","modified_gmt":"2026-10-26T07:48:08","slug":"hitmind-ai-crawler-check","status":"publish","type":"plugin","link":"https:\/\/wordpress.org\/plugins\/hitmind-ai-crawler-check\/","author":23590868,"comment_status":"closed","ping_status":"closed","template":"","meta":{"version":"1.0.0","stable_tag":"1.0.0","tested":"7.1.4","requires":"6.0","requires_php":"7.4","requires_plugins":null,"header_name":"HitMind \u2013 AI Crawler Check","header_author":"HitMind","header_description":"See what your robots.txt really tells GPTBot, ClaudeBot, PerplexityBot and 17 other AI crawlers, and catch blocked paths they can still reach.","assets_banners_color":"fdf9f8","last_updated":"2026-10-26 07:48:08","external_support_url":"","external_repository_url":"","donate_link":"","header_plugin_uri":"","header_author_uri":"https:\/\/hitmind.ai","rating":0,"author_block_rating":0,"active_installs":0,"downloads":66,"num_ratings":0,"support_threads":0,"support_threads_resolved":0,"author_block_count":0,"sections":["description","installation","faq","changelog"],"tags":{"1.0.0":{"tag":"1.0.0","author":"hitmind","date":"2026-10-26 07:48:08","revision":3765757}},"upgrade_notice":[],"ratings":[],"assets_icons":{"icon-128x128.png":{"filename":"icon-128x128.png","revision":3765757,"resolution":"128x128","location":"assets","locale":"","width":128,"height":128},"icon-256x256.png":{"filename":"icon-256x256.png","revision":3765757,"resolution":"256x256","location":"assets","locale":"","width":256,"height":256}},"assets_banners":{"banner-1544x500.png":{"filename":"banner-1544x500.png","revision":3765757,"resolution":"1544x500","location":"assets","locale":"","width":1544,"height":500},"banner-772x250.png":{"filename":"banner-772x250.png","revision":3765757,"resolution":"772x250","location":"assets","locale":"","width":772,"height":250}},"assets_blueprints":{},"all_blocks":[],"tagged_versions":["1.0.0"],"block_files":[],"assets_screenshots":{"screenshot-1.png":{"filename":"screenshot-1.png","revision":3765757,"resolution":"1","location":"assets","locale":"","width":1280,"height":900},"screenshot-2.png":{"filename":"screenshot-2.png","revision":3765757,"resolution":"2","location":"assets","locale":"","width":1280,"height":900},"screenshot-3.png":{"filename":"screenshot-3.png","revision":3765757,"resolution":"3","location":"assets","locale":"","width":1280,"height":900}},"screenshots":{"1":"The trap: blocked paths still open to named AI crawlers, and the lines that close them.","2":"Every AI crawler: what it is for, whether it follows robots.txt, the group it reads, and the verdict for your home page.","3":"The Site Health check."}},"plugin_section":[],"plugin_tags":[264107,246479,247966,12753,186],"plugin_category":[55],"plugin_contributors":[288606],"plugin_business_model":[],"class_list":["post-391148","plugin","type-plugin","status-publish","hentry","plugin_tags-ai-bots","plugin_tags-ai-crawlers","plugin_tags-gptbot","plugin_tags-robots-txt","plugin_tags-seo","plugin_category-seo-and-marketing","plugin_contributors-hitmind","plugin_committers-hitmind"],"banners":{"banner":"https:\/\/ps.w.org\/hitmind-ai-crawler-check\/assets\/banner-772x250.png?rev=3765757","banner_2x":"https:\/\/ps.w.org\/hitmind-ai-crawler-check\/assets\/banner-1544x500.png?rev=3765757","banner_rtl":false,"banner_2x_rtl":false},"icons":{"svg":false,"icon":"https:\/\/ps.w.org\/hitmind-ai-crawler-check\/assets\/icon-128x128.png?rev=3765757","icon_2x":"https:\/\/ps.w.org\/hitmind-ai-crawler-check\/assets\/icon-256x256.png?rev=3765757","generated":false},"screenshots":[{"src":"https:\/\/ps.w.org\/hitmind-ai-crawler-check\/assets\/screenshot-1.png?rev=3765757","caption":"The trap: blocked paths still open to named AI crawlers, and the lines that close them."},{"src":"https:\/\/ps.w.org\/hitmind-ai-crawler-check\/assets\/screenshot-2.png?rev=3765757","caption":"Every AI crawler: what it is for, whether it follows robots.txt, the group it reads, and the verdict for your home page."},{"src":"https:\/\/ps.w.org\/hitmind-ai-crawler-check\/assets\/screenshot-3.png?rev=3765757","caption":"The Site Health check."}],"raw_content":"<!--section=description-->\n<p>A crawler follows exactly one group in your robots.txt: the one that names it. The moment you give GPTBot a group of its own, it stops reading everything under <code>User-agent: *<\/code>, including the paths you blocked there (RFC 9309, section 2.2.1).<\/p>\n\n<p>It is an easy mistake to make. In HitMind's 2026 study of the top 10,000 websites, 29.8% of the robots.txt files that block paths and name an AI crawler left those paths open to that crawler.<\/p>\n\n<p>AI Crawler Check reads your live robots.txt the way crawlers do and shows:<\/p>\n\n<ul>\n<li><strong>The trap, if you have it:<\/strong> every path blocked under <code>User-agent: *<\/code> that a named AI crawler can still fetch, and the exact lines to add to its group.<\/li>\n<li><strong>All 20 AI crawlers, one row each,<\/strong> from OpenAI, Anthropic, Perplexity, Google, Apple, Meta, Amazon, Common Crawl, ByteDance and DuckDuckGo: what each one is for (AI answers that can link to you, model training, or other), whether its operator says it follows robots.txt, which group it reads, and whether it can fetch your home page.<\/li>\n<li><strong>Where to change the file:<\/strong> a real file in your site root, the one WordPress generates (and which SEO plugin adds to it), Cloudflare's managed robots.txt, or a WordPress install in a subfolder.<\/li>\n<li><strong>Lines crawlers skip,<\/strong> such as a misspelled directive.<\/li>\n<li><strong>A Site Health check,<\/strong> so the trap shows up under Tools \u2192 Site Health without opening the plugin.<\/li>\n<li><strong>A box to try changes<\/strong> before you publish them.<\/li>\n<\/ul>\n\n<p>What it does not do:<\/p>\n\n<ul>\n<li>It never changes your robots.txt. Which crawlers to block is your call; the plugin shows what your file says and how to make it say what you mean.<\/li>\n<li>It does not block or allow any crawler, and adds nothing to your pages.<\/li>\n<li>It stores nothing, tracks nothing and needs no account.<\/li>\n<\/ul>\n\n<p>The rules engine is the one behind HitMind's free <a href=\"https:\/\/hitmind.ai\/tools\/robots-txt-generator\">robots.txt generator and checker<\/a> and its <a href=\"https:\/\/hitmind.ai\/research\/ai-crawler-access-2026\">AI crawler study<\/a>. It gives the same verdicts on all 5,539 robots.txt files from that study.<\/p>\n\n<h3>External services<\/h3>\n\n<p>The plugin requests only your own site's \/robots.txt. It does not contact any other server.<\/p>\n\n<p>Its page has two links to hitmind.ai, which open only when you click them:<\/p>\n\n<ul>\n<li>\"Run a free SEO audit\" opens HitMind's free audit with your domain filled in (the domain is part of the link). <a href=\"https:\/\/hitmind.ai\/terms\">Terms of service<\/a>, <a href=\"https:\/\/hitmind.ai\/privacy\">privacy policy<\/a>.<\/li>\n<li>The 29.8% link opens HitMind's AI crawler study.<\/li>\n<\/ul>\n\n<p>Both links carry utm_source=wordpress-plugin so HitMind can count visits from the plugin. Nothing else is sent.<\/p>\n\n<!--section=installation-->\n<ol>\n<li>Install and activate the plugin.<\/li>\n<li>Go to Tools \u2192 AI Crawler Check.<\/li>\n<\/ol>\n\n<!--section=faq-->\n<dl>\n<dt id=\"does%20this%20plugin%20block%20ai%20crawlers%3F\"><h3>Does this plugin block AI crawlers?<\/h3><\/dt>\n<dd><p>No. It reads your robots.txt and reports what it finds. It never writes to it.<\/p><\/dd>\n<dt id=\"why%20is%20a%20path%20i%20blocked%20still%20open%20to%20gptbot%3F\"><h3>Why is a path I blocked still open to GPTBot?<\/h3><\/dt>\n<dd><p>Because GPTBot has its own group in your robots.txt. A crawler reads only the group that names it and ignores <code>User-agent: *<\/code> completely once it has one. Add the listed lines to its group, or remove the group if you meant GPTBot to follow your general rules.<\/p><\/dd>\n<dt id=\"some%20crawlers%20say%20they%20ignore%20robots.txt.%20does%20blocking%20them%20help%3F\"><h3>Some crawlers say they ignore robots.txt. Does blocking them help?<\/h3><\/dt>\n<dd><p>The table marks crawlers whose operators say they don't follow robots.txt, such as Perplexity-User, which fetches pages when a person asks for them. Rules for them state your position but are not enforced. Stopping them takes server or firewall rules.<\/p><\/dd>\n<dt id=\"it%20says%20it%20could%20not%20read%20my%20robots.txt.\"><h3>It says it could not read my robots.txt.<\/h3><\/dt>\n<dd><p>The plugin asks your own site for \/robots.txt, from your own server. Some hosts block that, and some firewalls and CDNs answer with a challenge page. Paste your file into the box at the bottom of the page to check it instead.<\/p><\/dd>\n<dt id=\"does%20it%20send%20my%20data%20anywhere%3F\"><h3>Does it send my data anywhere?<\/h3><\/dt>\n<dd><p>No. The only request it makes is to your own site's \/robots.txt. See \"External services\" for the two links on its page.<\/p><\/dd>\n<dt id=\"wordpress%20runs%20in%20a%20subfolder%20on%20my%20site.\"><h3>WordPress runs in a subfolder on my site.<\/h3><\/dt>\n<dd><p>robots.txt always lives at the root of the domain, so the plugin checks that file and tells you it is not served by WordPress.<\/p><\/dd>\n\n<\/dl>\n\n<!--section=changelog-->\n<h4>1.0.0<\/h4>\n\n<ul>\n<li>First release.<\/li>\n<\/ul>","raw_excerpt":"See what your robots.txt really tells GPTBot, ClaudeBot, PerplexityBot and 17 other AI crawlers, and catch blocked paths they can still reach.","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/wordpress.org\/plugins\/wp-json\/wp\/v2\/plugin\/391148","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wordpress.org\/plugins\/wp-json\/wp\/v2\/plugin"}],"about":[{"href":"https:\/\/wordpress.org\/plugins\/wp-json\/wp\/v2\/types\/plugin"}],"replies":[{"embeddable":true,"href":"https:\/\/wordpress.org\/plugins\/wp-json\/wp\/v2\/comments?post=391148"}],"author":[{"embeddable":true,"href":"https:\/\/wordpress.org\/plugins\/wp-json\/wporg\/v1\/users\/hitmind"}],"wp:attachment":[{"href":"https:\/\/wordpress.org\/plugins\/wp-json\/wp\/v2\/media?parent=391148"}],"wp:term":[{"taxonomy":"plugin_section","embeddable":true,"href":"https:\/\/wordpress.org\/plugins\/wp-json\/wp\/v2\/plugin_section?post=391148"},{"taxonomy":"plugin_tags","embeddable":true,"href":"https:\/\/wordpress.org\/plugins\/wp-json\/wp\/v2\/plugin_tags?post=391148"},{"taxonomy":"plugin_category","embeddable":true,"href":"https:\/\/wordpress.org\/plugins\/wp-json\/wp\/v2\/plugin_category?post=391148"},{"taxonomy":"plugin_contributors","embeddable":true,"href":"https:\/\/wordpress.org\/plugins\/wp-json\/wp\/v2\/plugin_contributors?post=391148"},{"taxonomy":"plugin_business_model","embeddable":true,"href":"https:\/\/wordpress.org\/plugins\/wp-json\/wp\/v2\/plugin_business_model?post=391148"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}