WordPress.org

Ready to get started?Download WordPress

Forums

BulletProof Security
[resolved] Linkchecker and other legit bots are broken (58 posts)

  1. AITpro
    Member
    Plugin Author

    Posted 1 year ago #

    Nope that did not work either. So I am at a dead end again and will revisit this another time. Whatever that facebook script is doing something about it must be seen as a threat to BPS. The error really is a 403 and not a 206 being logged as a 403 on the website end of it. When i check the linked image on facebook itself the image is displayed so back at square 1 - something about how the Header info is trying to be retrieved violates some security rule in BPS. The 206 error appears to happen anyway on WordPress and non-WordPress sites so my only concern is just getting rid of the nuisance and not worrying about what the issue is about the 206 error on the facebook end of it is.

    >>>>>>>>>>> 403 GET or Other Request Error Logged - February 8, 2013 - 12:04 pm <<<<<<<<<<<
    REMOTE_ADDR: 69.171.247.113
    Host Name: 69.171.247.113
    HTTP_CLIENT_IP:
    HTTP_FORWARDED:
    HTTP_X_FORWARDED_FOR:
    HTTP_X_CLUSTER_CLIENT_IP:
    REQUEST_METHOD: GET
    HTTP_REFERER:
    REQUEST_URI: /aitpro-blog/wp-content/themes/AITpro/images/aitpro-logo-footer.png
    QUERY_STRING:
    HTTP_USER_AGENT: facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
  2. wordpressmike
    Member
    Posted 1 year ago #

    From the rewrite log, do you know which line of the htaccess file is causing the 403 redirect?

  3. AITpro
    Member
    Plugin Author

    Posted 1 year ago #

    Oh you are talking about RewriteLogLevel. ;) My Host does not allow this and testing the facebook script cannot be done on XAMPP. So if your host allows this then give it a whirl. ;)

  4. AITpro
    Member
    Plugin Author

    Posted 1 year ago #

    pending further testing

  5. wordpressmike
    Member
    Posted 1 year ago #

    My development install that I can do rewrite logging on doesn't have traffic from facebook, so I can't help with logs.

  6. AITpro
    Member
    Plugin Author

    Posted 1 year ago #

    Yeah I have a similar situation. I have a Dedicated Server that I was not using so I am trying to get it setup enough to do the testing on it. Will be a few more days or so...

  7. AITpro
    Member
    Plugin Author

    Posted 1 year ago #

    Still looking at this one. I set up RewriteLogging and it does not tell me anything useful. ARGH. I have a couple of other ideas that i will be trying tomorrow.

  8. AITpro
    Member
    Plugin Author

    Posted 1 year ago #

    ARGH! So the facebook debugger is always going to return a 206 Response. That is just what happens. I confirmed this by doing several tests and I get a 206 Response no matter what I did.

    The debugger only requests the first 40KB of your page - so the 206 is expected (well, it's expected if only part of the document was returned but i guess some servers return it for any request with a Range: header)

    It shouldn't affect your ability to have the tags read correctly and the metadata populated when sharing a link on Facebook

    http://stackoverflow.com/questions/9807771/facebook-debugger-response-206

    The only thing i have to work with is the User Agent since nothing else is being logged. So I will try commenting out all User Agent filters and see what happens. nothing like trying to figure out something blind. LOL

    RewriteCond %{HTTP_USER_AGENT} (havij|libwww-perl|wget|python|nikto|curl|scan|java|winhttp|clshttp|loader) [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} (%0A|%0D|%27|%3C|%3E|%00) [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} (;|<|>|'|"|\)|\(|%0A|%0D|%22|%27|%28|%3C|%3E|%00).*(libwww-perl|wget|python|nikto|curl|scan|java|winhttp|HTTrack|clshttp|archiver|loader|email|harvest|extract|grab|miner) [NC,OR]
  9. AITpro
    Member
    Plugin Author

    Posted 1 year ago #

    I am very tempted at this point just to do something like this with the 403.php code since everything actually is working fine and the issue is just a nuisance issue.

    if ($_SERVER['REQUEST_METHOD'] != 'POST' && $_SERVER['HTTP_USER_AGENT'] != 'facebookexternalhit') {
    ...
    ...
    ...
  10. AITpro
    Member
    Plugin Author

    Posted 1 year ago #

    And strstr would be needed to capture all fb bot versions. This should work fine. Will test it and see what happens.

    if ($_SERVER['REQUEST_METHOD'] != 'POST' && strstr($_SERVER['HTTP_USER_AGENT'] != 'facebookexternalhit') ) {
    ...
    ...
    ...
  11. AITpro
    Member
    Plugin Author

    Posted 1 year ago #

    Actually this code is better for several reasons. testing...

    $facebookbot = 'facebookexternalhit';
    
    if ($_SERVER['REQUEST_METHOD'] != 'POST' && !strpos($_SERVER['HTTP_USER_AGENT'], $facebookbot) ) {
    ...
    ...
    ...
  12. AITpro
    Member
    Plugin Author

    Posted 1 year ago #

    wow ok so that does not work either.

  13. AITpro
    Member
    Plugin Author

    Posted 1 year ago #

    oh ok i see what the issue was. This does work. So i will add this in the next BPS release. You can of course add this code now if you want to the /bulletproof-security/403.php file.

    if ($_SERVER['REQUEST_METHOD'] != 'POST' && !preg_match('/facebookexternalhit(.*)/s', $_SERVER['HTTP_USER_AGENT'], $matches) ) {
    ...
    ...
    ...
  14. wordpressmike
    Member
    Posted 1 year ago #

    I don't think the solution should be to continue to block Facebook but not log the blocking. That sounds like a recipe for trouble.

  15. AITpro
    Member
    Plugin Author

    Posted 1 year ago #

    The facebook bot is not blocked - it never was blocked in the first place - image retreival was and is still happening perfectly. The problem is that something in the facebook externalhit_uatext.php script is not working correctly / not coded correctly, which then triggers a 403 error to be logged. The 206 error that is occurring with the facebook developers debugger tool is another coding problem and has nothing to do with BPS, WordPress or anything else externally. It also works fine, but the 206 error is misleading and sent me on a wild goose chase.

    So let me recap everything here.

    1. facebook image retrieval is working perfectly fine. It has been working fine from day one. Nothing is being blocked.

    2. The 206 error can be disregarded because this error occurs when you check any type of site and in every condition - it is just the way it is. it works, but gives the impression that something is wrong on the site that is being checked. What is wrong is that something is wrong with the debugger tool itself since this problem occurs on any site you check - HTML, WordPress, ASP, etc etc etc etc.....

    3. The ONLY real problem that exists is that there is a nuisance issue - 403 errors are being logged unnecessarily due to whatever is not working correctly / not coded correctly in the facebook externalhit_uatext.php script.

    So in summary there is ONLY 1 problem - the nuisance 403 errors that are being logged.

    Solution: Prevent the nuisance 403 errors from being logged.

    Case Closed

  16. wordpressmike
    Member
    Posted 1 year ago #

    Wow that is an amazing analysis.

  17. AITpro
    Member
    Plugin Author

    Posted 1 year ago #

    It is what it is. I'm only sorry I wasted so much time mucking around with this issue when I knew right from the get go this was just going to be a nuisance issue. LOL

    In any case, I am satisfied that the best solution is just to get rid of the nuisance issue after running 200 tests or so and confirming all is well.

  18. boybawang
    Member
    Posted 1 year ago #

    I'm running into the same problem too, and glad to see there's a fix!
    Where in the 403.php file should we put that line of code? I'm running BPS pro v 5.6.1. I have not made any modifications to that file yet.
    Thanks

  19. AITpro
    Member
    Plugin Author

    Posted 1 year ago #

    You can edit the /bulletproof-security/403.php file with the WordPress Plugins Editor.

    1. Click the Plugins Menu link >>> Editor.
    2. Select plugin to edit: BulletProof Security.
    3. Click the /bulletproof-security/403.php link.
    4. Scroll down in that 403.php file until you see this code...

    if ($_SERVER['REQUEST_METHOD'] != 'POST') {

    ...and modify the code to this...

    if ($_SERVER['REQUEST_METHOD'] != 'POST' && !preg_match('/facebookexternalhit(.*)/s', $_SERVER['HTTP_USER_AGENT'], $matches) ) {
  20. AITpro
    Member
    Plugin Author

    Posted 1 year ago #

    Oh just noticed you have the Pro version.

    You will need to turn off AutoRestore/Quarantine before doing the editing steps and will be looking for this code in the 403.php file...

    if ($_SERVER['REQUEST_METHOD'] != 'POST' && $options['bps_pfw_TMode'] != 'On') {

    ...and modify the code to this...

    if ($_SERVER['REQUEST_METHOD'] != 'POST' && $options['bps_pfw_TMode'] != 'On' && !preg_match('/facebookexternalhit(.*)/s', $_SERVER['HTTP_USER_AGENT'], $matches) {

    Then go to the AutoRestore page and click the wp-content Backup Files button.

    Or you can make the code changes with AutoRestore/Quarantine On and then when the 403.php file is sent to Quarantine after you have modified it you can use the Quarantine Restore File option to restore the modified 403.php file back to the /bulletproof-security plugin folder.

    FYI - you can also use the BPS Pro File Manager and All-purpose File Editor to edit the 403.php file instead of the WordPress Plugin Editor.

  21. boybawang
    Member
    Posted 1 year ago #

    Thanks, will try it out!

  22. AITpro
    Member
    Plugin Author

    Posted 1 year ago #

    oops i just realized that the coding in the BPS .47.9 403.php file was not updated and still has older coding in that file and the 400.php and 404.php files need to be updated. I will be releasing BPS .48 today that will have the updated code to prevent the facebook bot 403 errors from being logged anymore.

    if ($_SERVER['REQUEST_METHOD'] != 'POST' && !preg_match('/facebookexternalhit(.*)/s', $_SERVER['HTTP_USER_AGENT'], $matches) ) {
  23. AITpro
    Member
    Plugin Author

    Posted 1 year ago #

    BPS .48 has been released which contains the new 403.php code to not log the facebook bot errors. Resolved.

  24. boybawang
    Member
    Posted 1 year ago #

    Thanks for the fix! I'll give you feedback once I try it out. I'm out of town, so it'll most likely be next week sometime.

  25. boybawang
    Member
    Posted 1 year ago #

    Where can I find the .48 release of BPS with the above fix? I assumed that the update would automatically be indicated in my list of plugins, but I don't see it.

  26. AITpro
    Member
    Plugin Author

    Posted 1 year ago #

    You said you have BPS Pro 5.6.1 and not BPS .48 correct? The next release of BPS Pro will have this in it. For now you can just do the manual edit for BPS Pro above. Thanks.

  27. boybawang
    Member
    Posted 1 year ago #

    Ah, I understand now. Yes, I have 5.6.1. (pro edition). Thanks!

  28. boybawang
    Member
    Posted 1 year ago #

    Just wanted to give you a heads up - so far the replacement code for BPS pro works great!

Topic Closed

This topic has been closed to new replies.

About this Plugin

About this Topic