author here. If/when I get around to doing an update of these tests there are a lot of companies missing (flywheel, mediatemple, dreamhost, etc). Pantheon added it while I was working on this, so I didn't even get a chance to consider them for this go. I will definitely keep them in mind though.
Could you please at least update this version of the article with a link to the demo page that you blitz'd? Or, at least one public link to a webpagetest.org result that shows the number of requests, grades and total amount of data downloaded per page requested?
This is a great writeup and a real gem among a sea of junk hosting reviews out there, but I am having a hard time couching these results against my current service because blitz.io, and webpagetest.org results obviously vary to a HUGE degree depending the site / page tested and how it has been optimized.
My regular testing includes webpagetest.org and blitz.io as well, so I just need a little more of a frame of reference to decide whether or not I want to use one of those coupon codes and make a switch. Thanks!
EDIT: nope. That's was supposed to be a table of the webpagetest.org results :/ its 3am here and I need to get some sleep and be awake again in a few hours. I will have to get back to you on a nicer formatted version.
Thanks. That's actually more than enough to help me understand and compare. For completeness you may want to mention in your original review that you used a default wordpress install for the tests which is ~250-500kB downloaded and listed as "Bytes In" in a webpagetest.org result. Maybe you did and I missed it.
I started my tests with a default install too, but found that the actual sites I deal with are in the 1-2MB range so I imported "dummy content" I found somewhere to get more representative data and test different stacks of my own creation.
I decided to go with a $40/mo un-managed VPS 6 months ago partly because managed would have been hundreds of dollars more per month and partly because I wanted to learn how to do it myself. After approx 6mo of testing and optimizing various stacks with similar results to these managed hosts I have to say they represent a tremendous value. Configuration management, back ups, maintenance, security and disaster recovery of a high performance stack is a lot of work and I am not sure if I want to keep doing it.
That said, I could probably start my own managed hosting company that would compete well at this point!
I mentioned it was the default install but maybe that wasn't clear enough. I changed them all to 2014 theme too to give some parity. Real sites get heavier and heavier. When I load tests later they were on more realistic sites with some popular plugins enabled and dummy content. It's hard to make a 'representative' site though. Each install I've worked with has it's own set of plugins and feature-creep, it's hard to benchmark something like that and say it scales across everything.
It definitely is a lot of work managing your own sites and these managed providers certainly fit a niche. You wouldn't be the first to jump in and start offering it yourself. Pressable has a bunch of code shared on GitHub if you were serious about it too.
If you ever are interested in writing about your experience optimizing WP, I was thinking of doing an article like that as well. Maybe you could do a guest post? kevin at reviewsignal is my email if you're interested.
Interesting benchmarks. I am really impressed with the depth you have gone with these tests. It will be great to see newer companies included in that list too like cloudways.com and jumpstarter.io.
Pantheon, Cloudways and Jumpstarter are three great platforms with a potential to beat a lot of higher players in this industry.
Comments
I'm curious how pantheon would have done in this test as it is the new kid on the block. https://www.getpantheon.com/wordpress
author here. If/when I get around to doing an update of these tests there are a lot of companies missing (flywheel, mediatemple, dreamhost, etc). Pantheon added it while I was working on this, so I didn't even get a chance to consider them for this go. I will definitely keep them in mind though.
Could you please at least update this version of the article with a link to the demo page that you blitz'd? Or, at least one public link to a webpagetest.org result that shows the number of requests, grades and total amount of data downloaded per page requested?
This is a great writeup and a real gem among a sea of junk hosting reviews out there, but I am having a hard time couching these results against my current service because blitz.io, and webpagetest.org results obviously vary to a HUGE degree depending the site / page tested and how it has been optimized.
My regular testing includes webpagetest.org and blitz.io as well, so I just need a little more of a frame of reference to decide whether or not I want to use one of those coupon codes and make a switch. Thanks!
Let's see if this works.
WebPageTest.org – Dulles,VA Result URL WebPageTest.org – Miami, FL Result URL WebPageTest.org – Denver, CO Result URL WebPageTest.org –Los Angeles, CA Result URL 1.894s http://www.webpagetest.org/result/140217_6C_J3P/ 2.035s http://www.webpagetest.org/result/140217_DQ_HF4/ 2.381s http://www.webpagetest.org/result/140217_66_H6X/ 1.648s http://www.webpagetest.org/result/140217_JY_GRT/ 0.850s http://www.webpagetest.org/result/140304_FV_6XQ/ 0.961s http://www.webpagetest.org/result/140304_RG_6Z9/ 1.056s http://www.webpagetest.org/result/140304_MH_6ZH/ 0.665s http://www.webpagetest.org/result/140304_RN_70Y/ 1.245s http://www.webpagetest.org/result/140217_XW_J0Q/ 0.950s http://www.webpagetest.org/result/140217_KG_HW9/ 1.419s http://www.webpagetest.org/result/140217_N8_J39/ 0.924s http://www.webpagetest.org/result/140217_Z1_J7G/ 0.940s http://www.webpagetest.org/result/140217_QR_JT7/ 1.208s http://www.webpagetest.org/result/140217_XH_JJ0/ 1.229s http://www.webpagetest.org/result/140217_ER_JE6/ 0.671s http://www.webpagetest.org/result/140217_DW_JB9/ 0.642s http://www.webpagetest.org/result/140217_0K_JYE/ 1.174s http://www.webpagetest.org/result/140217_CB_K2X/ 1.721s http://www.webpagetest.org/result/140217_1Q_K61/ 0.981s http://www.webpagetest.org/result/140217_A8_KNH/ 1.073s http://www.webpagetest.org/result/140217_NR_M5G/ 1.327s http://www.webpagetest.org/result/140217_RM_M1H/ 1.682s http://www.webpagetest.org/result/140217_5P_KVQ/ 1.353s http://www.webpagetest.org/result/140217_ED_KPJ/ 0.732s http://www.webpagetest.org/result/140217_C4_M93/ 0.725s http://www.webpagetest.org/result/140217_1W_MGX/ 1.216s http://www.webpagetest.org/result/140217_9Q_MKK/
0.812s http://www.webpagetest.org/result/140217_X5_NH5/ 1.235s http://www.webpagetest.org/result/140217_RX_NJ1/ 1.060s http://www.webpagetest.org/result/140217_ND_NP4/ 1.080s http://www.webpagetest.org/result/140217_C1_NV3/ 0.924s http://www.webpagetest.org/result/140218_91_HS4/ 1.083s http://www.webpagetest.org/result/140218_MA_HSG/ 1.460s http://www.webpagetest.org/result/140218_B3_HTE/ 0.748s http://www.webpagetest.org/result/140218_VB_HW8/ 0.616s http://www.webpagetest.org/result/140219_EE_RAE/ 1.021s http://www.webpagetest.org/result/140219_CV_S2J/ 1.516s http://www.webpagetest.org/result/140219_HH_SKQ/ 1.116s http://www.webpagetest.org/result/140219_28_SV6/
EDIT: nope. That's was supposed to be a table of the webpagetest.org results :/ its 3am here and I need to get some sleep and be awake again in a few hours. I will have to get back to you on a nicer formatted version.
Thanks. That's actually more than enough to help me understand and compare. For completeness you may want to mention in your original review that you used a default wordpress install for the tests which is ~250-500kB downloaded and listed as "Bytes In" in a webpagetest.org result. Maybe you did and I missed it.
I started my tests with a default install too, but found that the actual sites I deal with are in the 1-2MB range so I imported "dummy content" I found somewhere to get more representative data and test different stacks of my own creation.
I decided to go with a $40/mo un-managed VPS 6 months ago partly because managed would have been hundreds of dollars more per month and partly because I wanted to learn how to do it myself. After approx 6mo of testing and optimizing various stacks with similar results to these managed hosts I have to say they represent a tremendous value. Configuration management, back ups, maintenance, security and disaster recovery of a high performance stack is a lot of work and I am not sure if I want to keep doing it.
That said, I could probably start my own managed hosting company that would compete well at this point!
I mentioned it was the default install but maybe that wasn't clear enough. I changed them all to 2014 theme too to give some parity. Real sites get heavier and heavier. When I load tests later they were on more realistic sites with some popular plugins enabled and dummy content. It's hard to make a 'representative' site though. Each install I've worked with has it's own set of plugins and feature-creep, it's hard to benchmark something like that and say it scales across everything.
It definitely is a lot of work managing your own sites and these managed providers certainly fit a niche. You wouldn't be the first to jump in and start offering it yourself. Pressable has a bunch of code shared on GitHub if you were serious about it too.
If you ever are interested in writing about your experience optimizing WP, I was thinking of doing an article like that as well. Maybe you could do a guest post? kevin at reviewsignal is my email if you're interested.
Flywheel is amazing, you should really add them to the test.
http://getflywheel.com/
Next round, definitely
Interesting benchmarks. I am really impressed with the depth you have gone with these tests. It will be great to see newer companies included in that list too like cloudways.com and jumpstarter.io.
Pantheon, Cloudways and Jumpstarter are three great platforms with a potential to beat a lot of higher players in this industry.