WordPress.org

Forums

BBQ: Block Bad Queries
[resolved] "The User Agent" part of the plugin won't work for me and botsvsbrowsers.com (10 posts)

  1. iframe
    Member
    Posted 11 months ago #

    Hello,

    Thanks for your great plugin.

    I was thinking about adding some nasty bots to your $user_agent_array. Before doing this, I was testing the current banned ones.

    Neither curl nor botsvsbrowsers.com gives 403, see the access log:

    curl:

    XX.XXX.XXX.XXX - - [22/Jun/2014:10:59:28 -0500] "GET / HTTP/1.1" 200 134707 "-" "casper"

    botsvsbrowsers.com:

    XX.XXX.XXX.XXX - - [22/Jun/2014:11:12:15 -0500] "GET / HTTP/1.1" 200 155818 "-" "zmeu"

    Please look into the matter.

    Thank you.

    https://wordpress.org/plugins/block-bad-queries/

  2. Jeff Starr
    Member
    Plugin Author

    Posted 11 months ago #

    Yeah, most proxy services change identifying headers like user-agent, so you're going to want to test using a service like http://www.hurl.it/ - there you can verify that BBQ is working correctly for the strings you mention.

  3. iframe
    Member
    Posted 11 months ago #

    Thanks, Jeff.

    hurl.it script didn't work and caused my browser crashing when I tried the website in question.

    However, when I picked another website, hurl.it script worked seamlessly, showing 403/200 when I activated/deactivated BBQ. Even my curl worked in unison with hurl.it

    Thanks again.

    M.

  4. iframe
    Member
    Posted 11 months ago #

    Wait a minute, Jeff.

    I have found and you confirmed that BBQ "The User Agent" part doesn't protect the website in question behind the proxy.

    We have direct hits from banned bots.

    There are enough proxies all over the world to make BBQ "The User Agent" part useless.

    Is there any workaround?

    Thank you.

    M.

  5. Jeff Starr
    Member
    Plugin Author

    Posted 11 months ago #

    I have found and you confirmed that BBQ "The User Agent" part doesn't protect the website in question behind the proxy.

    Actually, that is incorrect. BBQ does block malicious requests coming from proxy servers, and if the proxy is reporting a user agent that is blocked by BBQ, it should be blocked as well. In other words, BBQ will deny any request that reports a user agent that is included in the blocked-user-agents array.

    There are enough proxies all over the world to make BBQ "The User Agent" part useless.

    Lol, it may seem that way, but again, incorrect. As explained previously, BBQ denies requests from some of the worst user agents, regardless of whether the requests are made from behind a proxy.

    Updated to add: to block tough proxies, check out this technique:

    http://perishablepress.com/block-tough-proxies/

  6. iframe
    Member
    Posted 11 months ago #

    Sorry, Jeff.

    There are tools.

    I can't agree with you as long as curl with BBQ banned user-agents dumps everything from the site like BBQ doesn't exist.

    The access log proves that the banned user agent has reached the site and hasn't been denied.

  7. Jeff Starr
    Member
    Plugin Author

    Posted 11 months ago #

    Let's take a closer look at this. Here is the actual user-agent array to which you refer:

    $user_agent_array = apply_filters( 'user_agent_items', array( 'binlar', 'casper', 'cmswor', 'diavol', 'dotbot', 'finder', 'flicky', 'nutch', 'planet', 'purebot', 'pycurl', 'skygrid', 'sucker', 'turnit', 'vikspi', 'zmeu' ) );

    As you can see, there is no "curl" user agent on the list, so requests that include curl in the user agent are not blocked by BBQ. So if the curl request is using the default user agent (e.g., curl/1.2.3), it is not blocked by BBQ, by design. Otherwise, if the curl request sets a custom user agent using something like CURLOPT_USERAGENT, then BBQ will block only if the specified user agent is in the previously stated array.

  8. iframe
    Member
    Posted 10 months ago #

    Sure, it's easy to add 'curl' to array and check it out.

    What I found that when I clear cache (WP Super Cache) and send curl, it gives 403.

    curl -I http://xxxx
    HTTP/1.1 403 Forbidden

    As soon as a site being viewed in a browser and a cached file has been created, it gives 200.

    curl -I http://xxxx
    HTTP/1.1 200 OK

    I was able to duplicate such pattern on other sites running WP Super Cache as many times as I tried.

    I played with another site without WP Super Cache at all, 'curl' was blocked all the time.

    I removed 'curl' from the array and continued sending requests using
    '--user-agent <agent string>' to check the rest of $user_agent_array.

    Sure enough, BBQ blocks whatever is in the array.

    curl -I --user-agent "casper" http://xxxx
    HTTP/1.1 403 Forbidden

    Jeff, could you please explain why a cached file makes such a difference like there is no BBQ?

    Thank you.

  9. iframe
    Member
    Posted 10 months ago #

    Removed all BBQ instances.

    Problem solved.

    Unsubscribed.

  10. Jeff Starr
    Member
    Plugin Author

    Posted 10 months ago #

    Sorry for the delay on this! Thatnks for the reminder :)

    I'm not a cache expert, but would guess that it's because the pages are static, so some scripts aren't run (e.g., BBQ).. again, not an expert on caching but that would be my best guess. I hope it helps!

Reply

You must log in to post.

About this Plugin

  • BBQ: Block Bad Queries
  • Frequently Asked Questions
  • Support Threads
  • Reviews

About this Topic

Tags