{"id":18164964,"date":"2024-11-27T16:29:51","date_gmt":"2024-11-27T16:29:51","guid":{"rendered":"https:\/\/wordpress.org\/support\/topic\/blocking-crawler\/"},"modified":"2024-11-27T16:29:51","modified_gmt":"2024-11-27T16:29:51","slug":"blocking-crawler","status":"publish","type":"topic","link":"https:\/\/wordpress.org\/support\/topic\/blocking-crawler\/","title":{"rendered":"Blocking &#8220;crawler&#8221;"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">I seem to be having many sites that are bleeding bandwidth (upwards of 20GB a month and more) that aren&#8217;t getting caught by blackhole&#8230; in Awstats, it&#8217;s listed as &#8220;crawler&#8221; (when I tried adding a modsec rule for that, it seemed to be blocking multiple bots, so I&#8217;m not sure whether or not the Awstats info is reliable.)  In any case, while I do get occassional notices from the plugin that a bot has gotten blocked, it seems to miss a lot. Is the only protection based on a bot following a link it&#8217;s told not to? Is there something I&#8217;m missing in how to set this up effectively?<\/p>\n","protected":false},"template":"","class_list":["post-18164964","topic","type-topic","status-publish","hentry","topic-tag-crawler"],"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/wordpress.org\/support\/wp-json\/wp\/v2\/topic\/18164964","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wordpress.org\/support\/wp-json\/wp\/v2\/topic"}],"about":[{"href":"https:\/\/wordpress.org\/support\/wp-json\/wp\/v2\/types\/topic"}],"version-history":[{"count":0,"href":"https:\/\/wordpress.org\/support\/wp-json\/wp\/v2\/topic\/18164964\/revisions"}],"wp:attachment":[{"href":"https:\/\/wordpress.org\/support\/wp-json\/wp\/v2\/media?parent=18164964"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}