Support » Plugin: Broken Link Checker » Bug Report: Solution to 416 errors, 502 errors and other errors with https

  • Resolved gmcinnes

    (@gmcinnes)


    Hi guys:

    In modules/checkers/http.php lines 226-246 there’s an attempt to optimize cURL requests for checking link targets:

    $nobody = !$use_get; //Whether to send a HEAD request (the default) or a GET request

    $parts = @parse_url($url);
    if( $parts['scheme'] == 'https' ){
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); //Required to make HTTPS URLs work.
    curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
    $nobody = false; //Can't use HEAD with HTTPS.
    }

    if ( $nobody ){
    //If possible, use HEAD requests for speed.
    curl_setopt($ch, CURLOPT_NOBODY, true);
    } else {
    //If we must use GET at least limit the amount of downloaded data.
    $request_headers[] = 'Range: bytes=0-2048'; //2 KB
    }

    //Set request headers.
    if ( !empty($request_headers) ) {
    curl_setopt($ch, CURLOPT_HTTPHEADER, $request_headers);
    }

    Note that it uses a GET request for https and a HEAD request for http. Note also that it applies a Range header to the GET request.

    I’ve confirmed that a) HEAD requests work fine in https. I’m not sure why the comment on line 232 claims it doesn’t. Maybe it’s a historical thing? and b) GET requests with Range headers break lots of pages, causing them to be marked as false positives. They tend to fail with 416 Range not satisfiable errors if they don’t have the right bytes for the hardcoded range, or 502 Bad Gateway, if the site’s reverse proxy doesn’t know what to do with the Range request etc, etc. I’m sure there are more cases I haven’t identified.

    Here’s a quick snippet that demonstrates the issue. It’s a https request. HEAD works fine, GET works fine. GET with Range breaks.


    $nobody = !$use_get; //Whether to send a HEAD request (the default) or a GET request

    $parts = @parse_url($url);
    if( $parts['scheme'] == 'https' ){
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); //Required to make HTTPS URLs work.
    curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
    $nobody = false; //Can't use HEAD with HTTPS.
    }

    if ( $nobody ){
    //If possible, use HEAD requests for speed.
    curl_setopt($ch, CURLOPT_NOBODY, true);
    } else {
    //If we must use GET at least limit the amount of downloaded data.
    $request_headers[] = 'Range: bytes=0-2048'; //2 KB
    }

    //Set request headers.
    if ( !empty($request_headers) ) {
    curl_setopt($ch, CURLOPT_HTTPHEADER, $request_headers);
    }

    Let me know if you want me to submit a patch. I’m not sure how to go about it.

    https://wordpress.org/plugins/broken-link-checker/

Viewing 12 replies - 1 through 12 (of 12 total)
  • Oops. Wrong paste of the snippet there. Should be:

    <?php

    function get() {
    $ch = curl_init(“https://creativecommons.org/licenses/”);
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); //Required to make HTTPS URLs work.
    curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
    return $ch;
    }
    function get_with_range() {
    $ch = curl_init(“https://creativecommons.org/licenses/”);
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); //Required to make HTTPS URLs work.
    curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
    $request_headers[] = ‘Range: bytes=0-2048’;
    curl_setopt($ch, CURLOPT_HTTPHEADER, $request_headers);
    return $ch;
    }

    function head() {
    $ch = curl_init(“https://creativecommons.org/licenses/”);
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); //Required to make HTTPS URLs work.
    curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
    curl_setopt($ch, CURLOPT_NOBODY, true);
    return $ch;
    }

    function test($fn) {
    $ch = call_user_func($fn);
    curl_exec($ch);
    $info = curl_getinfo($ch);
    curl_close($ch);
    var_dump($info);
    }

    test(‘head’);
    test(‘get’);
    test(‘get_with_range’);
    ?>

    I second this!

    I also get many false positives with https. Makes the plugin useless in its current state.

    Agreed. I am seeing the same issue. Will be turning off the plug-in until it works.

    Plugin Author Vladimir Prelovac

    (@freediver)

    Yes please submit a patch to vladimir@prelovac.com

    The plugin will soon be github too.

    I was about to +1 this thread, then saw Vladimir’s post. Can’t wait 🙂

    I’m looking forward to this as well.

    For sure. I’m buzzed by false positive on SSL.

    Ok. Here’s a patch to be applied to modules/checkers/http.php.

    Unfortunately I don’t know the implications this could have on load levels for others. As written after this patch, the code should only make a GET request as a retry if a HEAD request fails. But now it GET’s the whole amount rather than only the first 2048 bytes, so it’s possible it could be downloading much more.

    However, my intuition is that a) That retry shouldn’t happen in very many cases and b) even if it does, since it’s not downloading assets etc like a browser would, but just the HTML, the impact should be pretty negligable. Unfortunately I have no way to test it in the wild for everyone, but it fixes things in my environment with no measurable impact. Something to look out for if you use it though.

    --- http.php.old	2015-10-15 12:29:54.000000000 -0400
    +++ http.php	2016-01-05 10:41:09.000000000 -0500
    @@ -228,16 +228,12 @@
             $parts = @parse_url($url);
             if( $parts['scheme'] == 'https' ){
             	curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); //Required to make HTTPS URLs work.
    -            curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
    -            $nobody = false; //Can't use HEAD with HTTPS.
    +            curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
             }
    
             if ( $nobody ){
             	//If possible, use HEAD requests for speed.
     			curl_setopt($ch, CURLOPT_NOBODY, true);
    -		} else {
    -			//If we must use GET at least limit the amount of downloaded data.
    -			$request_headers[] = 'Range: bytes=0-2048'; //2 KB
     		}
    
     		//Set request headers.

    @gmcinnes I should modify it myself or wait for update?

    @lotlog. No idea. Vlad has the patch, but I’m not sure how long it will take him to check it out and apply it, if he agrees. I hope it’s soon.

    For my client’s sake, I’d prefer not to be spending time working on my own fork.

    Plugin Author Vladimir Prelovac

    (@freediver)

    The sage version of the patch is up today, enjoy the new version.

Viewing 12 replies - 1 through 12 (of 12 total)
  • The topic ‘Bug Report: Solution to 416 errors, 502 errors and other errors with https’ is closed to new replies.