Block some spiders

Resolved jose
(@camperlife)

10 years, 3 months ago

Hello;
Since two days ago I’m receiving lots of Error logs notifications like these ones:

>>>>>>>>>>> 403 GET or Other Request Error Logged – 20 enero, 2014 – <<<<<<<<<<<
REMOTE_ADDR: 81.52.143.16
Host Name: crawl-b03-s1.voilabot.orange.fr
SERVER_PROTOCOL: HTTP/1.1
HTTP_CLIENT_IP:
HTTP_FORWARDED:
HTTP_X_FORWARDED_FOR:
HTTP_X_CLUSTER_CLIENT_IP:
REQUEST_METHOD: GET
HTTP_REFERER:
REQUEST_URI: /
QUERY_STRING:
HTTP_USER_AGENT: Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com)

>>>>>>>>>>> 403 GET or Other Request Error Logged – 20 enero, 2014 – <<<<<<<<<<<
REMOTE_ADDR: 199.30.20.24
Host Name: msnbot-199-30-20-24.search.msn.com
SERVER_PROTOCOL: HTTP/1.1
HTTP_CLIENT_IP:
HTTP_FORWARDED:
HTTP_X_FORWARDED_FOR:
HTTP_X_CLUSTER_CLIENT_IP:
REQUEST_METHOD: GET
HTTP_REFERER:
REQUEST_URI: /robots.txt
QUERY_STRING:
HTTP_USER_AGENT: msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)

>>>>>>>>>>> 403 GET or Other Request Error Logged – 20 enero, 2014 – <<<<<<<<<<<
REMOTE_ADDR: 199.30.20.24
Host Name: msnbot-199-30-20-24.search.msn.com
SERVER_PROTOCOL: HTTP/1.1
HTTP_CLIENT_IP:
HTTP_FORWARDED:
HTTP_X_FORWARDED_FOR:
HTTP_X_CLUSTER_CLIENT_IP:
REQUEST_METHOD: GET
HTTP_REFERER:
REQUEST_URI: /el-coloso-en-llamas/
QUERY_STRING:
HTTP_USER_AGENT: msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)

What can I do to block them.

http://wordpress.org/plugins/bulletproof-security/

Viewing 9 replies - 1 through 9 (of 9 total)

Plugin Author AITpro
(@aitpro)

10 years, 3 months ago

403 means blocked/Forbidden so they are already being blocked. The Security Log logs both blocked things: hackers, spammers, miners, harvesters, scrapers and also is a diagnostic tool to check for anything legitimate being blocked.

BPS does not block the msnbot by default / does not have any security rules that block the msnbot so you must be using some additional .htaccess rules/custom .htaccess code if the msnbot is being blocked.

Thread Starter jose
(@camperlife)

10 years, 3 months ago

The only added code I can see is written like this:

——————————————————-
# REQUEST METHODS FILTERED
# This filter is for blocking junk bots and spam bots from making a HEAD request, but may also block some
# HEAD request from bots that you want to allow in certains cases. This is not a security filter and is just
# a nuisance filter. This filter will not block any important bots like the google bot. If you want to allow
# all bots to make a HEAD request then remove HEAD from the Request Method filter.
# The TRACE, DELETE, TRACK and DEBUG request methods should never be allowed against your website.
RewriteEngine On
RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK|DEBUG) [NC]
RewriteRule ^(.*)$ – [F,L]
——————————————–
There’s also added code into the ‘Custom code brute force page protection” box
# BRUTE FORCE LOGIN PAGE PROTECTION
# Protects the Login page from SpamBots & Proxies
# that use Server Protocol HTTP/1.0 or a blank User Agent
RewriteCond %{REQUEST_URI} ^(/wp-login\.php|.*wp-login\.php.*)$
RewriteCond %{HTTP_USER_AGENT} ^$ [OR]
RewriteCond %{THE_REQUEST} HTTP/1\.0$ [OR]
RewriteCond %{SERVER_PROTOCOL} HTTP/1\.0$
RewriteRule ^(.*)$ – [F,L]

Thread Starter jose
(@camperlife)

10 years, 3 months ago

On the other hand Bing search engine is properly indexing my posts

Plugin Author AITpro
(@aitpro)

10 years, 3 months ago

Yep, it is going to be this type of thing in the link below. Several things are happening and one of those things is blocked, but nothing important is blocked.

http://wordpress.org/support/topic/does-bps-block-goodlegitimate-botsuser-agents?replies=7

Thread Starter jose
(@camperlife)

10 years, 3 months ago

OK; now this kind of entries have stopped for the moment so I’m going to check why I’m not receiving the Email alerts when the log files are deleted. It could be probably a plugin compatibility problem.
Thanks for your responses.
Plugin Author AITpro
(@aitpro)

10 years, 3 months ago
```
switch ($options['bps_security_log_size']) {
    case "256KB":
		if ( filesize($SecurityLog) >= 262144 && filesize($SecurityLog) < 524288 || filesize($SecurityLog) > 2097152) {
		if ( $options['bps_security_log_emailL'] == 'email') {
			if ( bps_Zip_Security_Log_File()==TRUE ) {
				bps_Email_Security_Log_File();
			}
		} elseif ( $options['bps_security_log_emailL'] == 'delete') {
			copy($SecurityLogMaster, $SecurityLog);
		}
		break;
		}
```
If you choose option:

Email Log & then delete log file you will receive your log file zipped in an email and the old log file will be replaced.

Delete log file only deletes the log file and does not send an email.

The logic is if a person does not want to receive their log files zipped in an email then they probably do not want to be emailed if they choose the delete log file option.
Thread Starter jose
(@camperlife)

10 years, 3 months ago

The Email Log & then delete log file was chosen but emails didn’t arrive.

Thread Starter jose
(@camperlife)

10 years, 3 months ago

And the same happens at ‘When an administrator logs in’ Email alerts.

Plugin Author AITpro
(@aitpro)

10 years, 3 months ago

Yes, that issue/problem happens when you have another plugin or theme that is taking over all email capability sitewide, which is of course a coding mistake in that plugin or theme. Or your Host Server does not allow the php mail() or the WordPress wp_mail() functions to be used on your website/Server. Or there is a configuration mistake in either the website’s php.ini file or the Server’s php.ini file.

Viewing 9 replies - 1 through 9 (of 9 total)

The topic ‘Block some spiders’ is closed to new replies.