Jetpack returning bad 302 HTTP format to varnish

  • Jasonheffner

    (@jasonheffner)


    12 years, 3 months ago

    I’m currently trying to resolve an issue specific to varnish and jetpack. When clicking on Authorize jetpack after enabling the plugin and being sent back to our site, varnish will present a 503 Guru mediation error. An example string looks like this..

    http://somesite.psu.edu/someblog/wp-admin/admin.php?page=jetpack&action=authorize&_wpnonce=cec3b377a1&code=vjBHdZkLIi&state=2

    If the page is reloaded with the same url the site will return as it should, and display “Your JetPack is already connected.” The initial 302 redirect is malformed and varnish won’t display it.

    Some info from varnish.. note varnish is simply passing the request. If I take varnish out it simply works since the browser will display the malformed 302 and redirect.

    12 RxRequest    c GET
   12 RxURL        c /someblog/wp-admin/admin.php?page=jetpack&action=authorize&_wpnonce=cec3b377a1&code=83VksrsrRa&state=2
   12 RxProtocol   c HTTP/1.1
   12 RxHeader     c Host: somesite.psu.edu
   12 RxHeader     c User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:18.0) Gecko/20100101 Firefox/18.0
   12 RxHeader     c Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
   12 RxHeader     c Accept-Language: en-US,en;q=0.5
   12 RxHeader     c Accept-Encoding: gzip, deflate
   12 RxHeader     c Cookie: __utma=165035810.2146825052.1359667423.1359667423.1359667423.1; __utmb=165035810.13.10.1359667423; __utmc=165035810; __utmz=165035810.1359667423.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); bp-activity-oldestpage=1; __qca=P0-1744595654-13596
   12 RxHeader     c Connection: keep-alive
   12 RxHeader     c X-Forwarded-For: 128.118.1.123
   12 VCL_call     c recv pass
   12 VCL_call     c hash
   12 Hash         c /someblog/wp-admin/admin.php?page=jetpack&action=authorize&_wpnonce=cec3b377a1&code=83VksrsrRa&state=2
   12 Hash         c somesite.psu.edu
   12 VCL_return   c hash
   12 VCL_call     c pass pass
   12 Backend      c 20 default_director web1
   12 FetchError   c http format error
   12 VCL_call     c error deliver
   12 VCL_call     c deliver deliver
   12 TxProtocol   c HTTP/1.1
   12 TxStatus     c 503
   12 TxResponse   c Service Unavailable
   12 TxHeader     c Server: Varnish
   12 TxHeader     c Content-Type: text/html; charset=utf-8
   12 TxHeader     c Retry-After: 5
   12 TxHeader     c Content-Length: 418
   12 TxHeader     c Accept-Ranges: bytes
   12 TxHeader     c Date: Thu, 31 Jan 2013 21:26:07 GMT
   12 TxHeader     c Age: 3
   12 TxHeader     c Connection: close
   12 TxHeader     c X-Cache: MISS
   12 Length       c 418
   12 ReqEnd       c 812364436 1359667565.384453297 1359667567.905995607 0.000034332 2.521497965 0.000044346

    I have ruled out the varnish config, timeouts, load, ssl, and other factors. I have recreated the problem on two separate multisite installs.

    http://wordpress.org/extend/plugins/jetpack/

Viewing 14 replies - 1 through 14 (of 14 total)
  • Plugin Author Jeremy Herve

    (@jeherve)

    Jetpack Mechanic 🚀

    12 years, 3 months ago

    While I am not really familiar with Varnish environments, other Jetpack users have reported issues in the past, and the following fix seems to solve the issue:
    http://usefulmix.com/wordpress-nginx-varnish-jetpack-xml_rpc-32700-glitch-fix/

    Could you have a look at it, and let me know if this helps?

    If it doesn’t, could you let me know your site URL? If you want it to remain private, you can also contact us via this form:
    http://en.support.wordpress.com/contact/?jetpack=needs-service

    Thread Starter Jasonheffner

    (@jasonheffner)

    12 years, 3 months ago

    That fix just bypasses varnish for your entire site. Unfortunately that is not going to work in our case as we have multiple users all connecting Jetpack from within our multisite installation. Yes, it does work for displaying the malformed response from the plugin as I have tried it, however, once varnish is proxying it spits it out.

    The call does actually go through and enables the plugin, and a refresh will create the correct response. I believe the initial response to the call is a malformed return and rejected by Varnish. I’ve seen some folks using a squid proxy to reconstruct the malformed response. I’m going to look more in depth as to why the response is malformed, unless you can help?

    fyi.. this started happening with the latest release and is the only page having problems on the site.

    Plugin Author Jeremy Herve

    (@jeherve)

    Jetpack Mechanic 🚀

    12 years, 3 months ago

    I’ll forward that issue to someone who has a better understanding of how Varnish works, if you don’t mind. 🙂

    We’ll get back to you as soon as we can address the issue.

    Thread Starter Jasonheffner

    (@jasonheffner)

    12 years, 3 months ago

    Thanks, We really appreciate it. I sent the site name and info to your support form citing this page.

    I also contacted a few with varnish to try and determine why it sees the response as malformed. I may have to do a capture in order to see the full details.

    Thread Starter Jasonheffner

    (@jasonheffner)

    12 years, 3 months ago

    I’ve managed to increase varnish settings to compensate for the large header that this particular request sends back to varnish. I still believe there is a coding issue with the session cookies jetpack creates in the header return.

    In varnish, I increased http_max_hdr and http_resp_size above the defaults to compensate.

    Plugin Contributor George Stephanis

    (@georgestephanis)

    12 years ago

    Howdy, Jason!

    Just trying to confirm one thing — do you have Varnish Caching turned on or off for your admin directories and xmlrpc.php?

    If it’s on, I’d recommend turning it off via something like …

    # Exclude the following
if (req.url ~ "^/([^/]*/?)wp-admin/(.*)" || req.url ~ "^/xmlrpc\.php" )  {
    return(pass);
}

    (I’m not a varnish or regex expert, but I believe that’s right)

    If it’s already off, then yeah, the headers are probably the issue.

    Thread Starter Jasonheffner

    (@jasonheffner)

    12 years ago

    It’s been turned off.. the tcpdump I sent them confirms it’s the headers. I can configure varnish to compensate for the large headers, but question how it was coded.

    if (req.url ~ “wp-(login|admin)|login” || req.url ~ “preview=true” || req.url ~ “xmlrpc.php”) {
    return (pass);
    }

    Thread Starter Jasonheffner

    (@jasonheffner)

    12 years ago

    George,

    Here was the header that call initiates and the dump I sent them a few months back. I think you can see the issue then and how they are checking states.

    Date: Fri, 01 Feb 2013 21:43:08 GMT
Server: Apache
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Cache-Control: no-cache, must-revalidate, max-age=0
Pragma: no-cache
X-Frame-Options: SAMEORIGIN
Set-Cookie: jetpackState[message]=authorized; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[message]=authorized; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[reactivated_modules]=vaultpress; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[error]=module_activation_failed; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[module]=widgets; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[reactivated_modules]=vaultpress%2Cwidgets; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[error]=module_activation_failed; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[module]=latex; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[reactivated_modules]=vaultpress%2Cwidgets%2Clatex; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[error]=module_activation_failed; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[module]=shortcodes; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[reactivated_modules]=vaultpress%2Cwidgets%2Clatex%2Cshortcodes; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[error]=module_activation_failed; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[module]=custom-css; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[reactivated_modules]=vaultpress%2Cwidgets%2Clatex%2Cshortcodes%2Ccustom-css; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[error]=module_activation_failed; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[module]=stats; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[reactivated_modules]=vaultpress%2Cwidgets%2Clatex%2Cshortcodes%2Ccustom-css%2Cstats; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[error]=module_activation_failed; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[module]=enhanced-distribution; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[reactivated_modules]=vaultpress%2Cwidgets%2Clatex%2Cshortcodes%2Ccustom-css%2Cstats%2Cenhanced-distribution; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[error]=module_activation_failed; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[module]=after-the-deadline; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[reactivated_modules]=vaultpress%2Cwidgets%2Clatex%2Cshortcodes%2Ccustom-css%2Cstats%2Cenhanced-distribution%2Cafter-the-deadline; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[error]=module_activation_failed; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[module]=sharedaddy; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[reactivated_modules]=vaultpress%2Cwidgets%2Clatex%2Cshortcodes%2Ccustom-css%2Cstats%2Cenhanced-distribution%2Cafter-the-deadline%2Csharedaddy; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[error]=module_activation_failed; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[module]=shortlinks; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[reactivated_modules]=vaultpress%2Cwidgets%2Clatex%2Cshortcodes%2Ccustom-css%2Cstats%2Cenhanced-distribution%2Cafter-the-deadline%2Csharedaddy%2Cshortlinks; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[error]=module_activation_failed; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[module]=gravatar-hovercards; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[reactivated_modules]=vaultpress%2Cwidgets%2Clatex%2Cshortcodes%2Ccustom-css%2Cstats%2Cenhanced-distribution%2Cafter-the-deadline%2Csharedaddy%2Cshortlinks%2Cgravatar-hovercards; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[error]=module_activation_failed; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[module]=subscriptions; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[reactivated_modules]=vaultpress%2Cwidgets%2Clatex%2Cshortcodes%2Ccustom-css%2Cstats%2Cenhanced-distribution%2Cafter-the-deadline%2Csharedaddy%2Cshortlinks%2Cgravatar-hovercards%2Csubscriptions; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[error]=module_activation_failed; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[module]=contact-form; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[reactivated_modules]=vaultpress%2Cwidgets%2Clatex%2Cshortcodes%2Ccustom-css%2Cstats%2Cenhanced-distribution%2Cafter-the-deadline%2Csharedaddy%2Cshortlinks%2Cgravatar-hovercards%2Csubscriptions%2Ccontact-form; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[error]=module_activation_failed; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[module]=post-by-email; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[reactivated_modules]=vaultpress%2Cwidgets%2Clatex%2Cshortcodes%2Ccustom-css%2Cstats%2Cenhanced-distribution%2Cafter-the-deadline%2Csharedaddy%2Cshortlinks%2Cgravatar-hovercards%2Csubscriptions%2Ccontact-form%2Cpost-by-email; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[error]=module_activation_failed; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[module]=mobile-push; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[reactivated_modules]=vaultpress%2Cwidgets%2Clatex%2Cshortcodes%2Ccustom-css%2Cstats%2Cenhanced-distribution%2Cafter-the-deadline%2Csharedaddy%2Cshortlinks%2Cgravatar-hovercards%2Csubscriptions%2Ccontact-form%2Cpost-by-email%2Cmobile-push; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[error]=module_activation_failed; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[module]=publicize; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[reactivated_modules]=vaultpress%2Cwidgets%2Clatex%2Cshortcodes%2Ccustom-css%2Cstats%2Cenhanced-distribution%2Cafter-the-deadline%2Csharedaddy%2Cshortlinks%2Cgravatar-hovercards%2Csubscriptions%2Ccontact-form%2Cpost-by-email%2Cmobile-push%2Cpublicize; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[error]=module_activation_failed; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[module]=notes; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[reactivated_modules]=vaultpress%2Cwidgets%2Clatex%2Cshortcodes%2Ccustom-css%2Cstats%2Cenhanced-distribution%2Cafter-the-deadline%2Csharedaddy%2Cshortlinks%2Cgravatar-hovercards%2Csubscriptions%2Ccontact-form%2Cpost-by-email%2Cmobile-push%2Cpublicize%2Cnotes; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[error]=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[module]=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/wp-admin; domain=somesite.psu.edu
Set-Cookie: jetpackState[php_errors]=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/wp-admin; domain=somesite.psu.edu
Location: https://somesite.psu.edu/wp-admin/admin.php?page=jetpack
Connection: close
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8
    Plugin Contributor George Stephanis

    (@georgestephanis)

    11 years, 12 months ago

    Oh dear me, yes, that’s a lot of cookies.

    Plugin Contributor George Stephanis

    (@georgestephanis)

    11 years, 12 months ago

    Well, if we can get the errors causing those resolved, the problem should go away.

    Are you getting anything in your PHP error log indicating why module activation could have failed?

    Thread Starter Jasonheffner

    (@jasonheffner)

    11 years, 12 months ago

    Module activation doesn’t fail. The call from the backend fails to be displayed to the client because varnish chokes on the header, and delivers a 503. Refreshing the page on the client side shows the module already enabled.

    I suspect a looping issue on module activation because of all the cookies created. jetpackState[error] and jetpackState[module] are set numerous times in the same header.

    I’m able to alleviate the error by increasing the http_max_hdr and http_resp_size in Varnish as mentioned above, but the initial cause is still there.

    Thread Starter Jasonheffner

    (@jasonheffner)

    11 years, 12 months ago

    George, there was nothing in the logs about any of the individual module_activation_failed errors from the cookies. The modules in jetpack do get enabled properly upon activation. As you can see some kind of loop happens when enabling them all in jetpack activation the first time on a site.

    Plugin Contributor Tim Moore

    (@tmoorewp)

    11 years, 11 months ago

    @jasonheffner: is your Varnish config set up to save cookies or no? It looks like attempts are being made to set the same cookie over and over and failing.

    Thread Starter Jasonheffner

    (@jasonheffner)

    11 years, 10 months ago

    @tim I don’t save cookies in varnish, but if a user is logged in the session is passed to apache and not cached. The above was a tcpdump between varnish and apache and was the only traffic I saw. Does jetpack make additional calls in process to set the cookies and hence generate those failed returns?

Viewing 14 replies - 1 through 14 (of 14 total)
  • The topic ‘Jetpack returning bad 302 HTTP format to varnish’ is closed to new replies.