iThemes Security (formerly Better WP Security)
Google-Bot and other search engine ID's for whitelisting (4 posts)

  1. hai3009
    Posted 2 years ago #


    is there a list of google, yahoo, msn bot IPO numbers out there?
    I often was blocked by google because the 404 errors blocked the search bot's.

    Kind Regards


  2. ianarmstrong
    Posted 2 years ago #


    Googlebot should never, ever, ever, ever, ever be locked out for any reason.

  3. acorath7
    Posted 2 years ago #

    One of the configurations tends to block the AdWords bot.

    Security > Tweaks > Filter Suspicious Query Strings

    When enabled, a rule gets added to the htaccess file with a plethora of conditions. One in particular is problematic...

    RewriteCond %{QUERY_STRING} ^.*(\[|\]|\(|\)|<|>|ê|"|;|\?|\*|=$).* [NC,OR]

    It's common practice with AdWords to dynamically add the search term keywords as a parameter to the URL. The problem is that the Bot doesn't use a keyword when checking to make sure that the URL is valid, so it ends up with something like this...


    Because the kw parameter is left empty by the bot, the last part of the expression is met and the rule is fired....

    RewriteRule ^(.*)$ - [F,L]

    As a result, Google receives a 403 forbidden, and the campaign is turned off.

  4. acorath7
    Posted 2 years ago #

    Also, it's worth noting that the behavior described above won't be apparent while you're logged into wordpress, due to another condition for the same rule...

    RewriteCond %{HTTP_COOKIE} !^.*wordpress_logged_in_.*$

    That little nugget had me scratching my head when Google was receiving 403's and I wasn't.

Topic Closed

This topic has been closed to new replies.

About this Plugin

About this Topic


No tags yet.