{"id":5324783,"date":"2014-09-24T01:12:42","date_gmt":"2014-09-24T01:12:42","guid":{"rendered":"https:\/\/wordpress.org\/support\/topic\/blocking-fake-googlebot\/"},"modified":"2016-08-22T01:50:25","modified_gmt":"2016-08-22T01:50:25","slug":"blocking-fake-googlebot","status":"closed","type":"topic","link":"https:\/\/wordpress.org\/support\/topic\/blocking-fake-googlebot\/","title":{"rendered":"Blocking &quot;fake&quot; Googlebot"},"content":{"rendered":"<p>I received an email from Google today that Googlebot couldn&#8217;t access my site:<\/p>\n<blockquote>\n<p>Over the last 24 hours, Googlebot encountered 67 errors while attempting to access your robots.txt. To ensure that we didn&#8217;t crawl any pages listed in that file, we postponed our crawl. Your site&#8217;s overall robots.txt error rate is 65.7%.<\/p>\n<\/blockquote>\n<p>I checked my site and found that Wordfence was blocking several IPs that when I ran WHOIS came back <em>AS<\/em> Google. Most frequently it was 173.194.99.134 <\/p>\n<p>To make it that my site isn&#8217;t overloading the CPU with the Googlebot trying to read my site, I have changed my settings under <strong>Firewall Rules<\/strong> to &#8220;Treat Google like any other Crawler&#8221; and unchecked &#8220;Block fake Google crawlers&#8221;<\/p>\n<p>I don&#8217;t know why Wordfence was blocking this IP but I don&#8217;t think its fake as I have watched the live activity on my site. I hope this can be corrected so I can change my settings back and block the crawlers that <em>are<\/em> fake.<\/p>\n<p>https:\/\/wordpress.org\/plugins\/wordfence\/<\/p>\n","protected":false},"template":"","class_list":["post-5324783","topic","type-topic","status-closed","hentry","topic-tag-block","topic-tag-crawler","topic-tag-googlebot","topic-tag-ip"],"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/wordpress.org\/support\/wp-json\/wp\/v2\/topic\/5324783","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wordpress.org\/support\/wp-json\/wp\/v2\/topic"}],"about":[{"href":"https:\/\/wordpress.org\/support\/wp-json\/wp\/v2\/types\/topic"}],"version-history":[{"count":0,"href":"https:\/\/wordpress.org\/support\/wp-json\/wp\/v2\/topic\/5324783\/revisions"}],"wp:attachment":[{"href":"https:\/\/wordpress.org\/support\/wp-json\/wp\/v2\/media?parent=5324783"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}