Support » Plugin: Blackhole for Bad Bots » Setup of Plugin – Confused, Please Assist

  • Resolved coffeecoffee

    (@coffeecoffee)


    Hi, I’m a bit lost as to how to follow these instructions in your plugin….

    Please help.

    When I open up the plugin’s settings, I see this:

    Add the following rules to your site’s robots.txt file:
    
    User-agent: *
    Disallow: /?blackhole

    And there’s a link in the instructions — the robots.txt file is linked to where you say, well, ‘robots.txt’…so I clicked it and I see:

    User-agent: *
    Disallow: /wp-admin/
    Allow: /wp-admin/admin-ajax.php

    Sooooo…..do I add your code, as written, in a separate block below what is already in my robots.txt file? Or do I add your code, as written, in a separate block above what is written there? And is there a carriage return (blank line, entire line that’s blank) between what is already written and your code if I do it this way?

    OR….uh…..haha….do I just add the one line that’s different as a separate line below the line that’s there? Like I could do something like…..this:

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /?blackhole
    Allow: /wp-admin/admin-ajax.php

    I HAVE NO IDEA what I’m doing, lol, and really don’t want to screw up my site or make the plugin ineffective…. 🙂

    Please note: I have WP Rocket installed (as well as Varnish cache on NGINX which is installed server-side by my host)….which is a cache plugin as you know. Just looked for info on how your plugin interacts with cache plugins and apparently that is an issue.

    Now that I saw a support thread on your plugin and caching being an issue that (still?) has to be resolved, I suppose my question is now two-fold:

    1.) How do I add the code as mentioned, properly….?
    2.) Is there other code I have to add to make this plugin compatible with my site and its WP Rocket cache (and Varnish cache)….?

    Thanks in advance for any help 🙂

    • This topic was modified 10 months, 1 week ago by  coffeecoffee.
Viewing 4 replies - 1 through 4 (of 4 total)
  • Plugin Author Jeff Starr

    (@specialk)

    Glad to help:

    1) Your third example looks fine to me, but I am not an SEO guy. If in doubt, either deactivate or hire an SEO wiz to help with your robots.txt. But yeah it /looks/ correct, based on all of the things you are saying.

    2) Here is a guide that explains how to configure Blackhole with supported caching plugins. Check out the section on WP Rocket for details on requirements, etc., should be enough to get you going.

    Let me know if any further questions, glad to help!

    • This reply was modified 10 months, 1 week ago by  Jeff Starr.

    Thanks!

    Uh, I have no idea what an SEO guy would do; not really an SEO question. Just wondering if your plugin will work with that code or not. Am I supposed to be most concerned about your plugin affecting SEO here? I hope not! All I want it to be is functional. You know how to make your own plugin work, I would assume….so, which of the three make it ‘work’…? Couldn’t care less about SEO with this right now. 🙂

    I only described two of the versions of code above. Did you not catch that? If that was hard to see then my apologies.

    The other two are not examples, but elements; the way they are to be used is in their description.

    So I suppose to get the point across properly/fully, I’ll post exactly what I meant instead of describing it. Here goes!

    VERSION 1:

    User-agent: *
    Disallow: /?blackhole
    
    User-agent: *
    Disallow: /wp-admin/
    Allow: /wp-admin/admin-ajax.php

    VERSION 2:

    User-agent: *
    Disallow: /wp-admin/
    Allow: /wp-admin/admin-ajax.php
    
    User-agent: *
    Disallow: /?blackhole

    VERSION 3:

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /?blackhole
    Allow: /wp-admin/admin-ajax.php

    I hope version 3 is still the best route, but I’m open to anything.

    As to ‘question 2’….I looked at that link (thanks for that). And…uh after looking at that page on the Blackhole Pro site……are you saying that we need to disable cacheing entirely for my site to use this plugin with WP Rocket? It appears so. Don’t know if you’ve found a fix for a compatibility issue with WP Rocket — God I hope you have. Because I need caching. And I love WP Rocket.

    Also: will the ‘cache helper’ plugin resolve any compatibility issues with WP Rocket? Or are we still told that it’s recommended to disable WP Rocket’s caching to use Blackhole for Bad Bots?

    Plugin Author Jeff Starr

    (@specialk)

    Version 3 looks correct to me.

    For your questions:

    “are you saying that we need to disable cacheing entirely for my site to use this plugin with WP Rocket?”

    Just the Page Caching option; otherwise your pages are static instead of dynamic, so plugins like Blackhole can’t work.

    “will the ‘cache helper’ plugin resolve any compatibility issues with WP Rocket?”

    Not sure if the Cache Helper will work for WP Rocket, I don’t think so but have not tested it. The Cache Helper was written/designed for WP Fastest Cache.

    “are we still told that it’s recommended to disable WP Rocket’s caching to use Blackhole for Bad Bots?”

    Whatever it says in that article I linked to is correct and the most current information. I believe the article says to disable the “page caching” feature only. All other features should work. But again, this is just from what other users have reported; I do not use any WP Rocket services.

    All 3 are correct and equivalent! The order and line spacing make no difference. Google “wordpress robots.txt” – there’s plenty of info online.

    Disallow: /wp-admin/
    Allow: /wp-admin/admin-ajax.php

    – is standard to prevent compliant bots from accessing /wp-admin/ but access to ajax is necessary for reasons I forget (and google wants it). You can look it up if you want to know the details. You still need to use robots.txt to control the behaviour of the ‘good’ bots, eg bingbot can eat up staggering amounts of bandwidth, at least it does on our site.

Viewing 4 replies - 1 through 4 (of 4 total)
  • The topic ‘Setup of Plugin – Confused, Please Assist’ is closed to new replies.