WordPress.org

Ready to get started?Download WordPress

Forums

iThemes Security (formerly Better WP Security)
[Plugin: Better WP Security] Better WP Blocking GoogleBot! (36 posts)

  1. ElleJG
    Member
    Posted 1 year ago #

    Using 3.4.3

    I see someone else here saying they are having problems with Analytics. I tried to do a Fetch as Google in Webmaster Tools on a new site, and got an error message. Returning a HTTP/1.1.418. All pages that had been indexed are now deindexed. Not good!!

    Contacted my host and they said it was this plugin. Of course I had deactivated all plugins EXCEPT this one while trying to find the problem. They suggested trying to whitelist Google bot but I see no way of doing that. Deactivated the plugin, tried fetch as google and it worked.

    The weird thing is I use this plugin on multiple sites. I've only checked one other - I'll be checking them all! - and Google is able to access that site. For the most part, I install all the same plugins on each site I set up.

    So a few things. WHY is this blocking Google and is there a way to whitelist Google if there is a problem?

    Oh, and one more thing. It appears that once you've enabled the banned default blacklist and added it, there is no way to disable this feature. That was the first thing I tried - before deactivating the plugin entirely - and once you disable the feature and save, it just shows as enabled again. I've tried this on my test site as well, and the same thing. No way to disable feature once enable.

    Thanks.

    http://wordpress.org/extend/plugins/better-wp-security/

  2. fulanoinc
    Member
    Posted 1 year ago #

    My site has flatlined after installing BWPS. Any word on what the problem is? Please help a guy out.

  3. ElleJG
    Member
    Posted 1 year ago #

    Did your site get deindexed? That's what happened to me. So you are the 3rd here to say you've had this plugin block GoogleBot in some way.

    And no, I have had no sort of answer, or even a request for further info to help diagnose the problem. I just deactivated BWPS and then Google was able to index my site again. Thankfully it was a new site and I was paying attention so not too much damage was done.

  4. The Hack Repair Guy
    Member
    Posted 1 year ago #

    I'm fairly sure the bad bots block list includes no IP's relating to Google so I can't imagine that aspect of Better WP Security is at fault.

    I've been using that block list for well over two years with some minor tweaks and not had a single report of googlebot related issues.

  5. borbou
    Member
    Posted 1 year ago #

    I got an emails from google webmaster tools saying "Google detected a significant increase in the number of URLs we were blocked from crawling due to authorization permission errors."

    The errors that have arrised was after i installed this plugin..

    BWPS blocks GoogleBot for sure..

    Any idea what to do next after deactivating BWPS ?

  6. nr56
    Member
    Posted 1 year ago #

    Exact same problem I had with WPBS. Blocked googlebots and destroyed my keyword rankings, traffic now 1/4 of what it used to be. Also multiple readers alerted me that they were being automatically logged in to one of my admin accounts after I installed this. Looks like a very useful plugin but unfortunately has a dangerous downside and should be avoided.

  7. The Hack Repair Guy
    Member
    Posted 1 year ago #

    I disagree. I've installed on hundreds of sites and none have reported any Googlebot blocking.

    I've seen just the opposite results (faster website, no effects on Google indexing, etc.).

    I think there is a lot more going on in this story.

    Before folks start pointing fingers let's use a little common sense and process to troubleshoot.

    First, there is a recommended ban list in Better WP Security.
    Please attempt to reinstall the plugin, then see:
    Ban -> un-check the box next to "Enable Default Banned List"

    Then re-run your Googlebot check once again with it off.

    (I've seen an number of apparently competitor posted anti-BWPS posts recently, hopefully this not another one)

  8. Basiq22
    Member
    Posted 1 year ago #

    I installed WPBS today on my new site and it blockacs google bot for me, do you guys find any solution for that?

    EDIT: i disabled WPBS and google bot works again.

  9. rbergeron81
    Member
    Posted 1 year ago #

    I believe this is a problem with the "filter request methods" option under the tweaks tab.

    Turn it off, and Google bots will work again.

  10. HCE
    Blocked
    Posted 1 year ago #

    Why does the "filter request methods" prevent Google bots from working?

  11. Handoko
    Member
    Posted 1 year ago #

    Not very sure, but these are what I know, hope it helps:

    - This plugin may block GoogleBot, it's true. I've ever seen GoogleBot and BingBot in my 404 error log.

    - Why? Due to not properly wrote theme and plugin I used, the bots were trying to looking for files (usually image) that do not exist in my website. So they generated 404 errors.

    - So, how to solve this issue? I put the a list of useful bots' IP in my 404 White List.

    - How to do it? Click the menu > Intrution Detection > 404 White List, then put the IPs there.

    - What are the IPs? Here are the IPs that ever listed in my 404 error log:
    157.55.32.107 (BingBot)
    157.55.33.44 (BingBot)
    157.55.34.183 (BingBot)
    65.55.52.95 (BingBot)
    65.55.213.192 (BingBot)
    66.249.73.144 (GoogleBot)
    66.249.76.1 (GoogleBot)
    66.249.76.5 (GoogleBot)
    66.249.76.5 (GoogleBot)

    - Should we enable Default Banned List or not? The Default Banned List can banned many bad bots, but also some might useful bots, so I do not enable it.

    - So far, I do not have any problem with Filter Request Methods enabled.

  12. The Hack Repair Guy
    Member
    Posted 1 year ago #

    404 error means page does not exist.

  13. Handoko
    Member
    Posted 1 year ago #

    If you enable Blacklist Repeat Offender on Intrusion Detection (404 Detection), it will automatically ban IPs that generate 404 errors after reach certain threshold. I did see Google's IPs in my list of the 404 log. It means if GoogleBot keep generating some more 404 errors, it will be banned.

  14. sopedro
    Member
    Posted 1 year ago #

    Hi,

    Maybe BWPS conflicts with some plugin.
    It conflicts with w3 total cache, you have to install total cache before bwps. If i install after there's no way to get total cache working.

  15. jdaviescoates
    Member
    Posted 1 year ago #

    Oh, and one more thing. It appears that once you've enabled the banned default blacklist and added it, there is no way to disable this feature. That was the first thing I tried - before deactivating the plugin entirely - and once you disable the feature and save, it just shows as enabled again. I've tried this on my test site as well, and the same thing. No way to disable feature once enable.

    I've noticed that too, on a completely clean install of wp and an up to date clean install of BWPS.

    Also, if you go and manually change .htaccess to revert the changes, BWPS dashboard doesn't pick that up and still thinks it is enabled (i.e. the line is still shown as green with no link next to it).

    The only way I could get it to reset was to delete and reinstall BWPS

  16. Handoko
    Member
    Posted 1 year ago #

    Oh, and one more thing. It appears that once you've enabled the banned default blacklist and added it, there is no way to disable this feature.

    Never knew about it. Anyone please start a new thread reporting this issue.

  17. erna.braat
    Member
    Posted 1 year ago #

    I have the same problem: This plugin blocks googlebots, and I never turned on Filter Request Methods. Besides that, after clicking on menu items I get blank pages with 'error' in the left corner. Deactivating this plugins was the only way to get back to the backend after logging in again.
    I am so sorry...this plugin seemed to be what I really need. Security and backup in one plugin. But it gives me and my clients much stress...

  18. Handoko
    Member
    Posted 1 year ago #

    It sounds your issue is 404 errors.

    Goto menu > Security > View Logs. You need to examine the 404 errors. I can think a very possible cause is, one of your plugin or theme is not properly written, it generate lots of 404 errors when visitor visits your page, so you or Google being blocked.

    Goto menu > Security > Intrusion Detection. You may need to set a higher value for the Error Threshold, or perhaps totally disable the Enable 404 Detection.

    Actually, too many 404 errors can mean you're under hacking attacks. This Better WP Security will try to block the attempts and show an 'error' message. But sometimes it can be happen due to not properly written plugins or theme, or wrong settings (permalinks for example).

  19. AITpro
    Member
    Posted 1 year ago #

    @Handoko - I don't normally butt in, but I see you saying this over and over in several different Threads - "I can think a very possible cause is, one of your plugin or theme is not properly written, it generate lots of 404 errors". I think that this is a bit misleading for folks.

    404 errors are a completely normal thing that occur with any website. You are always going to have 404 errors from time to time. The most common cause of 404 errors is that some other website that has created a "bad" link to your website, which is causing the unintentional 404 errors. Yes, you are correct about malicious/intentional 404 errors created to cause problems for another website, but the majority of 404 errors are just someone making a mistake.

    If someone moved a post or page and it is not automatically being handled already by WordPress then the site owner should create a RedirectMatch 301 htacess rule so that the 404 errors no longer occur.

    Examples:
    RedirectMatch 301 ^/old-category/my-old-post$ /new-category/my-new-post$1

    subfolder redirect
    RedirectMatch 301 ^/website-folder-name/tag/403$ http://www.example.com/website-folder-name/

    root site redirect
    RedirectMatch 301 ^/tag/best-wordpress$ http://www.example.com/

  20. Handoko
    Member
    Posted 1 year ago #

    Okay, perhaps I use the wrong words, I admit. I'm sorry.

    As you said, majority of 404 errors are just someone making a mistake. The possibility are:

    - The webmaster, move/delete a post/page. Or change the settings but without update (for example, using caching plugin you may need to clear the cache)
    - The plugin/theme writer hardcoded wp-content directory, so 404 errors may happens if you change it
    - The plugin/theme writer request a file (usually image) that is really not exist.

    I have some experiences, saw 404 errors that a plugin requesting a file that was not exist, and in other case it requesting a hardcoded wp-content directory. I contacted the authors using support forum, and yes, they fix them.

    404 errors are a completely normal thing that occur with any website.
    Yes, but if the user configures things properly and the authors being more careful writing the code, it should not happen. Except hacking attempts, trying to scan website's vulnerability (like timthumb).

    I'm not a security expert. I'm not try to defend myself. I do know the words I used might be a bit misleading. English is not my native language. I will more careful next time. Thanks for warning me.

  21. AITpro
    Member
    Posted 1 year ago #

    I was just concerned that folks would think that all 404 errors come from plugins and themes. Yep your are right that some definitely do come from plugins and themes, but most of them come from external sources. Like another website is linking to an old link on a site or just created a bad link to a site. So I was just pointing that out. I hope I did not offend you. That was not my intention. Just wanted to make sure folks are getting all the info regarding 404 errors. ;)

    I recently spent a month of ongoing cleanup of 404 errors because some site (no names) was grabbing my site URLs for the purpose of stats and ranking postings, but that sites code was fubar and was mixing up the URL's with different sites. Had to clean up over 4,000 404 errors over a month. Grrr.

  22. Handoko
    Member
    Posted 1 year ago #

    I didn't feel being offended. When I typed about plugin/theme authors could be the possibility, I really thought it could be misleading. It sounds like the authors are to be blamed. I really didn't mean so, I just want to suggest users to contact the authors as what I did, so the result: the better codes.

    Perhaps, I'm lacking experiences. Most of my 404 errors (except from hackers) are hardcoded wp-content, I changed my content directory. I ever deleted a post, I saw Google keep coming back every several weeks for it. So now, I won't delete or rename any post/page. That's why this kind of 404 errors (deleted/moved links) are very seldom on my sites.

    Thanks for pointing it out.

    We all here just want to share experiences hope can be useful for others. You did good jobs, your plugins are great. I ever thought to use it, but I prefer this Better WP Security. You mentioned BulletProof Security can work harmonious with Better WP Security, that's great unfortunately I don't want to install to many plugins which my slowdown or consume lots of my server resources.

  23. AITpro
    Member
    Posted 1 year ago #

    Ok good I thought I might have offended you. Thanks for letting me know I did not. :)

  24. erna.braat
    Member
    Posted 1 year ago #

    Thank you both Handoko and AITpro for sharing your knowledge. I use the plugin on multiple websites. I had this problem on 1 website; the blank pages with the little word 'error' in the left corner and the issue; Googlebot blocked.
    On this particular website I had also installed the plugin Google XML Sitemaps. Do you guys think that this causes a conflict?

  25. Handoko
    Member
    Posted 1 year ago #

    My suggestions are:

    - Whitelist Googlebot immediately. Goto menu > Security > Instrution Detection > 404 White List > put the IPs there. You may need use this link to check who is the IP:
    http://www.projecthoneypot.org/search_ip.php

    - If you see the word 'error', it can mean your IP has been blocked. You may need to (temporary) disable Blacklist Repeat Offender before you're banned from your website. Goto menu > Security > Instrution Detection > disable Blacklist Repeat Offender.

    - Do you use any caching plugin? Empty all the caches. We should always empty the caches after changing configurations, install/activate/deactivate plugin.

    - Did you recently move your website to another webhost? Or did just rename or delete any post/page? These may cause 404 errors because someone (or Google) may have link to the missing URL.

    - Did you change your wp-content directory (menu > Security > Content Directory)? This may cause 404 errors. You need to examine your logs to see if it your issue.

    - Examine your logs, goto menu > Security > View Logs. You need to study the pattern. What IPs appear most frequently? Where did they came from? What were they requesting? Are they human, crawlers or bots? Use ProjectHoneyPot to check:
    http://www.projecthoneypot.org/search_ip.php

    - If the IPs are bad (can be checked using ProjectHoneyPot), you may need to ban it. Goto menu > Security > Ban Users > Enable Banned Users and put the IPs in Ban Hosts.

    - Apple device users can cause 404 error. Are apple-touch-icon being mentioned in your log?

    It's hard to say what is the cause of your problem without examine your logs. It might be intrusion attacks from hackers or only a simple mistake which is nothing to worry. Recently, I did see bad bots faking tobe GoogleBots using IP spoofing technique.

    I use Google XML Sitemaps too, have no problem working with Better WP Security.

  26. erna.braat
    Member
    Posted 1 year ago #

    Thank you for your comprehensive manual!

    Sorry, it took a while before I dared to install the plugin WP Better again.
    Unfortunately, now I have the same problem on a second website: white page with ' error ' in the top left corner.

    I've looked at the Logs and put my own IP address on the White list. But it seems to me it is not intended to put IP addresses of Google bots on the White list... I can't find them and moreover, I think they change all the time.

    Yes, in my log 'apple-touch-icon' was being mentioned, but what should I do about that? It is quiet a job to keep an eye on all the bad IP addresses and to put them on a black list manually...

    I do not use a Cache plugin.
    Where can I clean up any Cache?

    On this second website I never used Google XML Sitemaps, so fyi this plugin is not causing any problems in combination with Better WP Security.

  27. Handoko
    Member
    Posted 1 year ago #

    For the issue of "white page with error", you can find more info here:
    http://wordpress.org/support/topic/site-eror

    For apple-touch-icon issue, read here:
    http://wordpress.org/support/topic/receiving-so-many-site-lockout-notifications

    Normally you don't need to whitelist GoogleBots. You may need to do it if your website is having too many 404 errors. Having too many 404 errors may mean there is something wrong in your website, you should inspect and fix it. So whitelisting GoogleBots is not the best thing do do, but a quick temporary fix to avoid Google being banned from your website.

    If you don't use cache plugin, you don't need to do any cache cleaning. Almost all caching plugins do have a button for user to empty the cache.

    it took a while before I dared to install the plugin WP Better again.
    You may need these info:
    http://wordpress.org/support/topic/it-seems-very-risky-to-use-this-plugin

  28. tinozee
    Member
    Posted 1 year ago #

    Hi,

    First of all this is a really cool plugin. I have been getting the Googlebot IPs locked out as well. On lookup it seems this is a real google ip being blocked - 66.249.75.146.

    My setup is a copy of my site on a test acct., with a real domain pointed to it for accurate testing and dns config testing. I also have the WP install set to "Discourage search engines from indexing this site". Not sure if that settibng would come into play at all but I thought I would mention it.

    Once I can configure this plugin such that it will not interfere with googlebot I will use it. Is adding IPs to a white list the only mechanism besides disabling this feature?

    Thanks.

  29. Handoko
    Member
    Posted 1 year ago #

    Sorry, I'm not very understand, English is not my native language.

    You said you have a copy site for testing purpose, with a real domain pointed to it.

    - Why a real domain pointed to a test site
    - Why Googlebot visits you test site, even you have discourage the visits
    - Why you care Googlebot being lockout from your test site

    I personally think, for testing purposes you should not let any real domain (or webiste) pointing to the test sites. Because it may invite Google and others to crawl your page.

    If you have turned on "Discourage ...", Google should not visit the site. But if Google really visited your test site, you no need to care Google being blocked (locked out) from the test sites. Because if it really crawls the pages, it may have duplicate content problems (you said it is a copy of the 'real' site).

  30. tinozee
    Member
    Posted 1 year ago #

    Hi,

    Thanks for the response. Ahh the domain is disabled now and we use IP. The domain was needed for some complex DNS setup that we had to mimic and test in a test environment. That has been disabled so no worries.

    The only question I have is this -

    Is the white list the only way to allow the googlebot IP? I do not want to enable this plugin on my live site if this may happen. What if google bot gets a new IP? How will the plugin add it to the white list?

    I ask my question because I want to know the answer BEFORE I use this plugin on the live site. I think that is the obvious way to approach this.

Topic Closed

This topic has been closed to new replies.

About this Plugin

About this Topic

Tags

No tags yet.