Support » Plugin: iThemes Security (formerly Better WP Security) » Google-Bot and other search engine ID's for whitelisting

Viewing 3 replies - 1 through 3 (of 3 total)
  • ianarmstrong

    (@ianarmstrong)

    +1

    Googlebot should never, ever, ever, ever, ever be locked out for any reason.

    acorath7

    (@acorath7)

    One of the configurations tends to block the AdWords bot.

    Security > Tweaks > Filter Suspicious Query Strings

    When enabled, a rule gets added to the htaccess file with a plethora of conditions. One in particular is problematic…

    RewriteCond %{QUERY_STRING} ^.*(\[|\]|\(|\)|<|>|ê|”|;|\?|\*|=$).* [NC,OR]

    It’s common practice with AdWords to dynamically add the search term keywords as a parameter to the URL. The problem is that the Bot doesn’t use a keyword when checking to make sure that the URL is valid, so it ends up with something like this…

    http://domain/path/to/page/?kw=

    Because the kw parameter is left empty by the bot, the last part of the expression is met and the rule is fired….

    RewriteRule ^(.*)$ – [F,L]

    As a result, Google receives a 403 forbidden, and the campaign is turned off.

    acorath7

    (@acorath7)

    Also, it’s worth noting that the behavior described above won’t be apparent while you’re logged into wordpress, due to another condition for the same rule…

    RewriteCond %{HTTP_COOKIE} !^.*wordpress_logged_in_.*$

    That little nugget had me scratching my head when Google was receiving 403’s and I wasn’t.

Viewing 3 replies - 1 through 3 (of 3 total)
  • The topic ‘Google-Bot and other search engine ID's for whitelisting’ is closed to new replies.