Bad Behavior rejects Google crawls

michaellinder
(@michaellinder)

13 years, 5 months ago

My site’s always had great Google placement, but it took weeks of bad behavior by Bad Behavior to alienate Google which has dropped me from its listings. BB’s Michael Hampton admits his WordPress plug-in has a bug that rejects Google site crawlers as imposters.

“You claimed to be a major search engine, but you do not appear to actually be a major search engine” Bad Behavior repeatedly told Google crawls as documented in multiple view-as-Googlebot errors at Google Webmaster Tools.

“I’m pretty sure this is a bug in Bad Behavior, but I haven’t been able to isolate it yet,” emailed Hampton on November 10. “For the moment I suggest you discontinue using Bad Behavior until I find and squash this one.”

I’ve so far been unable to lure Google back to my pages after weeks of rejection. I’ve filed a request to reinstate my site, but Google says it’ll be weeks before anyone gets around to reading the plea or acting on it.

I strongly advise any Bad Behavior user to perform a view-as-Googlebot search in Webmaster tools to see whether Bad Behavior is rejecting Google searches.

Viewing 14 replies - 1 through 14 (of 14 total)

Uli
(@uli)

13 years, 5 months ago

EXACTLY! Took me weeks to figure out, but tracing back my precipitous drop from google crawls, BB 2.0.38 must have been the culprit.

The solution: delete Bad Behavior 2.0.38 and reinstall BB 2.0.37
from http://www.bad-behavior.ioerror.us/download/
That one works and doesn’t block Googlebots as checked from within Google Webmaster central (FETCH AS GOOGLEBOT)
Vijay Padiyar
(@vijaypadiyar)

13 years, 3 months ago
Not just Google, I see that it is even blocking Amazon crawls.

Today I saw this entry in my log, which was basically Amazon’s bot crawling a new post on my site:

67.202.14.30
ec2-67-202-14-30.compute-1.amazonaws.com

2011-01-10 19:35:14
```
Required header 'Accept' missing</strong>

GET /blog/2011/01/big-bazaar-shopping-nightmare?utm_source=feedburner&utm_medium=twitter&utm_campaign=Feed%3A+VijayPadiyarsWebsite+%28Vijay+Padiyar%27s+Website%29&utm_content=Twitter HTTP/1.1
Host: www.vijaypadiyar.in
Referer: http://twitter.com
User-Agent: Mozilla/5.0 Gecko/20100914 Firefox/3.6.10
```
67.202.14.30 is a clean IP and belongs to Amazon. I’m thinking whether to disable BB if this continues! For now I guess I’ll whitelist this IP.
Pieter Hartsook
(@hartsook)

13 years, 3 months ago

BB has rev’d to 2.0.39 – has anyone determined if this inappropriate blocking behavior has been fixed with the new version?

Vijay Padiyar
(@vijaypadiyar)

13 years, 3 months ago

Sorry, I should’ve mentioned that I’m using the latest version (2.0.39), and this is happening with the latest version.

I have sent an email to Amazon AWS regarding this. And it’s happening with multiple IP addresses of Amazon, everytime with the same error (Required header ‘Accept’ missing). Seems to be poor programming on part of Amazon!

Although on one instance, I saw that the request was blocked due to the IP being on BB’s blacklist!! Wonder how that happened.

Regards

Vijay Padiyar

error
(@error)

13 years, 3 months ago

Vijay, that request is not from an Amazon crawler; it is from a spammer or other malicious robot which rents virtual server space in Amazon’s cloud.

Michaellinder, the reason you were having trouble is you are using CloudFlare with a development version of Bad Behavior; your issue should be fixed in the latest development release. If you find that it isn’t, please reply to the email I sent you two months ago.

microberto
(@microberto)

12 years, 11 months ago

Just wanted to add that I was just logged into Google Webmaster Tools, and used the “Fetch as GoogleBot” tool to see what one of my pages looked like to G. And….. DENIED BY BAD BEHAVIOR.

If you have ANY interest in SEO (which you should), you should NOT use this plugin. Even if it blocks just a very small percentage of Google’s datacenters, it is not worth it!!!! My stronger sites could probably power through that, but my newer/weaker sites couldn’t. This particular site is not ranking for ANYTHING.

Deactivated and Deleted on all sites. I won’t be returning.

Watch out folks…
Pieter Hartsook
(@hartsook)

12 years, 11 months ago
MicroBerto, I followed your proceedure and hit my WordPress 3.1.2 site using the Google Webmaster Tools, “Fetch as GoogleBot”. I am running BB Version 2.0.43.

Contrary to your results I had success on the home page and a sub-page:
```
http://hartsook.com/portfolio   Web	Success	5/11/11 11:41 PM
http://hartsook.com/            Web	Success	5/11/11 11:40 PM
```
FWIW I am shared hosting at FatCow.com

Bottom line BB is NOT blocking Google Bot on my site.
Vijay Padiyar
(@vijaypadiyar)

12 years, 11 months ago

Googlebot doesn’t use just one single IP address. It has several IP addresses. IF what MicroBerto is saying is true, then perhaps Google is now using a new IP address which is not yet recognised by BB. Perhaps when hartsook tried the same thing, the Googlebot request might have been sent from a different IP address from what was used in MicroBerto’s case.

MicroBerto, I would ask you to go to Bad Behavior page (Tools > Bad Behavior) and check for the blocked entry there. The entry will be similar to what I have posted above.

You can check for entries with User-agent field as Googlebot or similar. Paste the entire entry here. That will help us see the IP address of the crawl request and the reason for it being blocked.

Regards

Vijay Padiyar

treydock
(@treydock)

12 years, 11 months ago

MicroBerto, what version are you running? Could you post the log output generated by Bad Behavior?

The only time Bad Behavior has blocked legitimate traffic on my installs has been due to a poorly configured Google appliance at my University. But that’s when the whitelist option is useful.

– Trey

treydock
(@treydock)

12 years, 11 months ago

MicroBerto, if you could tell us the IP then maybe it needs to be added to Bad Behavior.

These are the current IP ranges used to identify Google bots…
66.249.64.0/19
64.233.160.0/19
72.14.192.0/18
203.208.32.0/19

desertgeek
(@desertgeek)

12 years, 11 months ago

Version 2.0.43 works great and I’v been watching it VERY closely for the last 8 days. Get your http:BL Access Key.
and it performs it’s magic. Iv’e yet to see it block a LEGITIMATE SE crawler. PERIOD. NOTE that I said a SE crawler and NOT some data service crawling away or a bot that pretends to be a good guy. Another issue that bugs me is the notion that spam to your email address will increase from that 403 page. Not true.

I got all excitied when I saw one of those threat level 76 try it’s thing. YIKES! Never saw a log for a single IP/Visit that large. LOL 🙂

I hope this helps folks understand what the current version does.

faithmechanic
(@faithmechanic)

12 years, 11 months ago

BB was blocking Google on my site also. I receive excellent Google rankings within my content niche and then 3 days ago – Boom. I found I was receiving tons of crawl errors. I went into Fetch as Googlebot and I received an error for a recent page. The error code referenced Bad Behavior. Once BB was off my system, the error disappeared.

What is going on? My site – http://hopefaithprayer.com/

BB, we love you but please stop dissing Google! I was using Version: 2.0.43 and the latest WordPress.

microberto
(@microberto)

12 years, 10 months ago

Sorry guys, I stopped using the plugin and don’t want to risk enabling it. I’m an SEO and it’s simply not worth it in terms of risk/reward. Just run a honeypot and continually collect the good GoogleBot addresses using Webmaster Tools or something to that extent.

error
(@error)

12 years, 8 months ago

Sorry, but Google doesn’t tell me when they add IP address ranges. So I have to find out somehow. As far as I know, every Google address range in use is whitelisted in Bad Behavior and has been for several months (since 2.0.43). If you find Google is being blocked, you need to contact me and report it immediately, including the IP address that Google used. Posting to the forum will NOT get you action.