WordPress.org

Forums

iThemes Security (formerly Better WP Security)
Google-Bot and other search engine ID's for whitelisting (4 posts)

  1. hai3009
    Member
    Posted 1 year ago #

    Hi,

    is there a list of google, yahoo, msn bot IPO numbers out there?
    I often was blocked by google because the 404 errors blocked the search bot's.

    Kind Regards

    http://wordpress.org/plugins/better-wp-security/

  2. ianarmstrong
    Member
    Posted 1 year ago #

    +1

    Googlebot should never, ever, ever, ever, ever be locked out for any reason.

  3. acorath7
    Member
    Posted 1 year ago #

    One of the configurations tends to block the AdWords bot.

    Security > Tweaks > Filter Suspicious Query Strings

    When enabled, a rule gets added to the htaccess file with a plethora of conditions. One in particular is problematic...

    RewriteCond %{QUERY_STRING} ^.*(\[|\]|\(|\)|<|>|ê|"|;|\?|\*|=$).* [NC,OR]

    It's common practice with AdWords to dynamically add the search term keywords as a parameter to the URL. The problem is that the Bot doesn't use a keyword when checking to make sure that the URL is valid, so it ends up with something like this...

    http://domain/path/to/page/?kw=

    Because the kw parameter is left empty by the bot, the last part of the expression is met and the rule is fired....

    RewriteRule ^(.*)$ - [F,L]

    As a result, Google receives a 403 forbidden, and the campaign is turned off.

  4. acorath7
    Member
    Posted 1 year ago #

    Also, it's worth noting that the behavior described above won't be apparent while you're logged into wordpress, due to another condition for the same rule...

    RewriteCond %{HTTP_COOKIE} !^.*wordpress_logged_in_.*$

    That little nugget had me scratching my head when Google was receiving 403's and I wasn't.

Topic Closed

This topic has been closed to new replies.

About this Plugin

About this Topic

Tags

No tags yet.