Bots do not seem to be decreasing

  • Resolved Trinary Star

    (@trinary-star)


    1 week, 3 days ago

    Hello. I admire your work. So, I started to use this blackhole plugin on May 19, 2026 on one of my websites. It has been more than a month since then, but the number of bots coming to my website does not seem to be decreasing. Rather, it is increasing.

    First thing first. No caching is used. So that should not cause it.

    63 bots seem to have been trapped, all with iphone User-Agent strings.

    What is wrong? Or is this how this plugin should work? Or it is too early to evaluate the effectiveness of this plugin.

    The page I need help with: [log in to see the link]

Viewing 11 replies - 1 through 11 (of 11 total)
  • Plugin Author Jeff Starr

    (@specialk)

    1 week, 3 days ago

    Glad to help, and it’s a great question. Basically: there are many types of bots that target all sorts of things. This plugin does not block all bots, only the ones that disobey your site’s robots.txt rules. So that means, any bots that DO obey the rules are NOT blocked. And these days there are MANY bots that follow robots.txt properly.

    • This reply was modified 1 week, 3 days ago by Jeff Starr.
    Thread Starter Trinary Star

    (@trinary-star)

    1 week, 1 day ago

    But as you can see in my robots.txt, all bots except a few like Googlebot are disallowed on my blog. Do you mean most bots ignore that part of the robots txt directives?

    Plugin Author Jeff Starr

    (@specialk)

    1 week, 1 day ago

    “you mean most bots ignore that part of the robots txt directives?”

    It depends on many factors. Some sites get many disobedient bots, others fewer. There are many bots that do not obey robots.txt. Also there are many bots that do obey robots.txt.

    Thread Starter Trinary Star

    (@trinary-star)

    1 week, 1 day ago

    Then what about deleting the robots.txt altogether? Doesn’t that ban all bots except the ones this plugin specifically allows?

    Plugin Author Jeff Starr

    (@specialk)

    1 week, 1 day ago

    “Then what about deleting the robots.txt altogether?”

    Not recommended unless you really understand what you are doing. But it’s your site and there is no requirement for sites to provide a robots.txt file.

    “Doesn’t that ban all bots except the ones this plugin specifically allows?”

    The robots.txt file only bans as defined by the site owner. This plugin does not ban any bots per se, rather it forbids access to the hidden blackhole/trap link. So bots that obey robots rules will not follow the hidden link and fall into the trap.

    Plugin Author Jeff Starr

    (@specialk)

    1 week, 1 day ago

    Btw most of what we are discussing here is explained in the plugin docs, the home and installation pages, help tab on the settings page, etc. Just fyi in case you would like to read more details about how things work with the plugin, bad bots, robots.txt, and other related information.

    Thread Starter Trinary Star

    (@trinary-star)

    1 week ago

    If unwanted bots follow robots.txt directives for Googlebot, doesn’t this plugin prove ineffective?

    My robots.txt is as follows:

    ————————————-

    User-agent: Googlebot
    Disallow:
    Disallow: /*blackhole
    Disallow: /?blackhole

    (skipped)

    User-agent: *
    Disallow: /*blackhole
    Disallow: /?blackhole

    ————————————-

    What I meant is that instead of this kind of robots.txt, it may be better to get rid of unwanted bots whose User-Agent strings are known by directives in the .htaccess file. Googlebots can be blocked from getting into the blackhole with the .htaccess directives. Unwanted bots will get into the blackhole and then get banned by this plugin.

    What do you think?

    Plugin Author Jeff Starr

    (@specialk)

    1 week ago

    “If unwanted bots follow robots.txt directives for Googlebot, doesn’t this plugin prove ineffective?”

    That’s a pretty big “if”, and certainly not the case IRL. Keep in mind this plugin is free, so you are welcome to not use it if you have other ideas.

    “instead of this kind of robots.txt, it may be better to get rid of unwanted bots whose User-Agent strings are known by directives in the .htaccess file.”

    Yes absolutely, in fact I have put years of work into providing free Apache/.htaccess solutions at Perishable Press. For example here is my latest related work: Ultimate Block List to Stop AI Bots. Literally hundreds of other bot-fighting resources and information also available. Blackhole for Bad Bots is just one of them.

    Thread Starter Trinary Star

    (@trinary-star)

    1 week ago

    You have been very informative. I will surely study the above mentioned work of yours. Thank you.

    Thread Starter Trinary Star

    (@trinary-star)

    1 week ago

    However…there seems to be no way of writing such directives (something that tells Googlebot not to access blackhole directories) in an .htaccess file as blackhole directories are created dynamically.

    Plugin Author Jeff Starr

    (@specialk)

    1 week ago

    Yeah maybe, you would need to research further to find out, what’s possible etc. It’s not something that is provided by the plugin though.

    In any case, I am glad to have helped. Gonna go ahead and mark this thread as resolved. Feel free to post again with any further questions, etc. Thank you.

Viewing 11 replies - 1 through 11 (of 11 total)

You must be logged in to reply to this topic.