adding additional bots to block list

  • kimhardesty

    (@kimhardesty)


    9 years, 3 months ago

    I’ve noticed on my server logs (c-panel, AWStats) that I am still getting some access from unknown bots. I’m especially concerned with those top ones above googlebot. Should I be concerned with these? How do I determine their names and/or how to block them using Wordfence?

    Name / Hits(+hits on robots.txt) / Bandwidth / Last Visit
    Unknown robot (identified by ‘bot’ followed by a space or one of the following characters _+:,.;/\-) 1,665+29 38.60 MB 02 Jan 2017 – 06:06
    Unknown robot (identified by ‘spider’) 1,096+3 23.11 MB 02 Jan 2017 – 06:05
    Unknown robot (identified by ‘crawl’) 507+8 10.81 MB 02 Jan 2017 – 06:03
    Unknown robot (identified by empty user agent string) 477+1 51.12 MB 02 Jan 2017 – 06:01
    Googlebot 279+2 5.25 MB 02 Jan 2017 – 06:01
    Ruby script 244 4.80 MB 02 Jan 2017 – 06:03
    A PHP script 200 25.75 MB 02 Jan 2017 – 05:52
    Unknown robot (identified by ‘robot’) 78+2 1.95 MB 02 Jan 2017 – 06:04
    Unknown robot (identified by ‘bot’ preceded by a space or one of the following characters _+:,.;/\-) 20+22 1.54 MB 02 Jan 2017 – 05:17
    Yahoo Slurp 32+5 684.69 KB 02 Jan 2017 – 04:42
    Unknown robot (identified by hit on ‘robots.txt’) 0+10 712 Bytes 02 Jan 2017 – 03:11
    MSNBot-media 4 102.63 KB 02 Jan 2017 – 02:11
    Google Web Preview 2 25.55 KB 01 Jan 2017 – 09:27
    Unknown robot (identified by ‘discovery’)

Viewing 1 replies (of 1 total)
  • wfalaa

    (@wfalaa)

    9 years, 3 months ago

    Hi,
    I think you may be suffering from an increase in bandwidth because of certain bots, while there is no option to selectively block custom bots in Wordfence, but I’m pretty sure you can easily block any of these bots IPs from (Wordfence > Live Traffic) and it will get blocked for the period you set in “How long is an IP address blocked when it breaks a rule“, after that you can go to “Wordfence > Blocked IPs” to block it permanently, I should warn you that blocking IPs permanently isn’t recommend as these IPs are dynamically assigned to different machines and you might end up blocking a legit IP address after a while.

    Thanks.

Viewing 1 replies (of 1 total)

The topic ‘adding additional bots to block list’ is closed to new replies.