WordPress.org

Ready to get started?Download WordPress

Forums

W3 Total Cache
Using W3TC with non-Amazon CDN which is fully compliant with the S3 API (10 posts)

  1. Gwyneth Llewelyn
    Member
    Posted 1 year ago #

    Hi,

    I'm looking for a simple solution for using the CDN option with a non-Amazon solution which, however, is fully compatible with the S3 API.

    DreamHost is launching DreamObjects, which fully supports both the S3 and RackSpace (Swift) APIs. However, there seems to be no simple way to get W3TC to recognise a different, S3-compliant endpoint than Amazon's own.

    What I've seen so far is that each time a new CDN becomes popular, W3 Edge launches a new option on the CDN configuration page. That's great, but it could be even greater if it allows certain common APIs (S3 is probably the most popular among those) to be used with different endpoints.

    Short of hacking the code and replacing the call for Amazon with the domain name for DreamObject's API endpoint, is there a simpler way to do that?

    I suppose that using the CDN via FTP might, for now, do the trick (even though I don't know yet if that works as expected or not — I haven't tried), but it will lose the richness of using S3's API for uploading/updating content on the CDN.

    I think that all that is needed is a box with the endpoint API server, defaulting to Amazon's S3 (or RackSpace's), but being editable (so that I can change it to objects.dreamhost.com, in my case). That would be awesome! More and more CDN providers are relying on open source software like Ceph which exposes a fully compliant S3 or Swift API — it makes sense to easily change just the server name to take advantage of all those CDNs, instead of waiting until W3 Edge to add yet another option on the list of supported CDNs...

    Or am I totally missing the point here? Perhaps this is already possible on a "hidden" option somewhere?

    Thanks in advance for any input.

    http://wordpress.org/extend/plugins/w3-total-cache/

  2. Ipstenu-DH
    DreamHost Rep
    Posted 1 year ago #

    Right now, DreamObjects doesn't offer full CDN services. It's a 'Yet' thing, however, we're working on it!

    I think that all that is needed is a box with the endpoint API server, defaulting to Amazon's S3 (or RackSpace's), but being editable (so that I can change it to objects.dreamhost.com, in my case). That would be awesome!

    That would be awesome!

  3. Gwyneth Llewelyn
    Member
    Posted 1 year ago #

    Oh, thanks for the correction, Mika. I was not being very precise, because, in truth, I'm rather unfamiliar with Amazon's terms. It seems that what Amazon calls a CDN is CloudFront. What I had in mind was merely the S3 service, so, cloud-based storage, similar to what DreamHost is offering (in beta!) and exposing a compatible API.

    Here is what I did. After setting up a bucket on DreamHost and marking it as "public", I went to /wp-content/plugins/w3-total-cache/lib and edited S3.php on lines 819 and 1347, changing the hard-coded domain name s3.amazonaws.com to objects.dreamhost.com.

    Then go to /wp-content/plugins/w3-total-cache/lib/W3/Cdn/S3.php and do the same on line 339, and, for good measure, change it on /wp-content/plugins/w3-total-cache/lib/W3/Cdn/S3/Cf.php on line 95 as well. This should be enough.

    Then I went to the General tab for the W3TC plugin. I selected CDN and as a CDN Type Amazon Simple Storage Service. W3TC, at this point, complains that you need to fill in all the remaining data and push content to the cloud service.

    Then I went to the CDN tab and put my API key, secret, and bucket name. Did a test (clicking the button below) and it was green — the test had passed!

    Now came the next step: starting moving the initial content to the bucket. This is simply achieved by clicking a button on the messages that W3TC displays to remind you that everything needs to be pushed on the cloud. A popup appeared, saying that there were 382 objects to be pushed, and after pressing Start, it happily started to push objects over smoothly, with a green "all clear" status for each of them. So things seemed to be working!

    This actually worked quite well for a few minutes, with well over 200 objects placed in the bucket, until after a while I started getting red status saying: [503] Unexpected HTTP status. Aw! And it was running so smoothly. Intrigued, I went to the DreamHost control panel, to see what was inside the bucket so far... and got "Network Error. Try Again Later". So I did, waiting for a bit. After a few minutes, the Network Error disappeared and I could see all the objects which had been pushed so far. Yay!

    Back to WordPress and clicking "export media library" once more. W3TC correctly identified that a lot of objects were already on the cloud (yay!) and up-to-date, so it just pushed the missing ones over. Cool! Everything seems to work as expected!

    Then I proceeded to push the remaining bits over — wp-includes, theme files, custom files, and so forth. Often I get the dreaded network error again, and for a few minutes, nothing gets uploaded. So it takes some patience... upload as much as you can, watch for errors, wait a few minutes until the network error disappears (checking on the DreamHost control panel), and trying again. Eventually everything will be uploaded, sooner or later. After all, DreamObjects is still "beta" :)

    The end result: WordPress is now using DreamObjects using the S3 API to store all static files as seen by W3TC :) Cool stuff! Now I will very likely not be using this configuration for long. As DreamObjects are still in beta, this means that the nasty "Network errors" will simply fail to get anything retrieved from them while the situation goes on unattended. As said, sooner or later the Network Error will go away, and things will work as expected for a while.

    So, this sort of proves my point. It would be awesome if W3 Edge's developers would allow changing the endpoint without requiring it to be hacked. As more and more cloud storage providers use S3 (or RackSpace's) API as a standard, it makes little sense to "limit" S3 just to Amazon. It also doesn't make sense to keep adding more and more options each time a new provider becomes popular. All that is needed is a box to add a new option on the CDN tab to allow a different hostname as the API endpoint.

    In fact, I understand that this S3.php is some sort of a standard library. The constructor of the class S3 defaults to s3.amazonaws.com, but the constructor can be called with a different hostname, and the library doesn't need to be changed at all. I saw that the option page is generated from /wp-content/plugins/w3-total-cache/inc/options/cdn, but I couldn't (yet) find the place where the WP options for W3TC are stored (to add a new one, say, cdn.s3.endpoint or something like that, and make sure it gets correctly called from the place where the class S3 is invoked by W3TC. It seems to be only called at two places, /wp-content/plugins/w3-total-cache/lib/W3/Cdn/S3.php and /wp-content/plugins/w3-total-cache/lib/W3/Cdn/S3/Cf.php but of course I'm not sure. If I have nothing to do during the weekend, I might also change that, and see what happens, even though I have no clue how to get in touch with W3 Edge to review the changes and eventually incorporate them in their next release. For now, for people not worried about hacking things, the simple hack I've suggested works fine.

    I might move these instructions to the DreamHost wiki, too, since they might be useful for others... and post them in my blog as well, where I'm sure nobody will delete them :)

  4. Ipstenu-DH
    DreamHost Rep
    Posted 1 year ago #

    That was amazingly done! Yes, please bring it over to our wiki. I was holding off working on it for the DreamObjects plugin, since my tests showed that using DO as my CDN didn't help as much as I wanted due to the lack of true CDN support.

    There are other plugins and stand-alone apps that are starting to allow that info to be editable, because we're not the only company getting on board with the cloud like this :)

  5. Gwyneth Llewelyn
    Member
    Posted 1 year ago #

    Thanks, Mika :) Instructions were posted on the DreamHost wiki as well

  6. Gwyneth Llewelyn
    Member
    Posted 1 year ago #

    I'm unable to keep up with updating information on multiple sources, so, for now, assume that the DreamHost wiki will have the latest hacks :)

    I have also added a further nicety, using DH's justinlund's suggestion, and implemented CNAME redirection. This will allow users of CloudFlare, for example, to also cache requests not only coming from their WP installation, but also from DreamObjects. As an extra bonus, some page ranking algorithms, for some reason, pay attention that all requests come from the same base domain name (I don't know why that is deemed to be "better"). Using CNAMEs to point to DreamObjects allows that.

    So right now this means that there is a fully working solution for DreamObjects. Maybe it's time to pester W3 Edge to take a peek at my hacks, validate them, and incorporate it on the next release of W3TC <nudge, nudge, wink, wink> :-)

  7. Gwyneth Llewelyn
    Member
    Posted 1 year ago #

    Unfortunately, W3 Edge released a new version which still doesn't include non-Amazon, S3-compatible code, so it means hand-hacking everything once more.

  8. Frederick Townes
    Member
    Plugin Author

    Posted 1 year ago #

    Great suggestion. Will look at adding that for future versions.

  9. Gwyneth Llewelyn
    Member
    Posted 1 year ago #

    Glad to hear that :) In the mean time, the latest two or three releases of W3TC have changed so much that I was unable to get those hacks working again. So, sadly, I have to give up on using DreamHost's non-Amazon S3-compatible cloud-based storage.

    I'd be more than welcome to try it again if and when a future version of W3TC is released which makes the whole process easier!

  10. Ipstenu-DH
    DreamHost Rep
    Posted 1 year ago #

    Yikes, good to know. We're using it on DreamHost.com and have yet to upgrade.

Topic Closed

This topic has been closed to new replies.

About this Plugin

About this Topic