The post WordPress Hosting Providers Study: Web Performance & Scalability appeared first on LoadStorm.
]]>
When it comes to web performance, study after study has proven: fast and scalable wins the race. But with thousands of WordPress hosting providers, how do you know which one is fast and scalable?
That is where ReviewSignal.com comes in. Their business is all about helping people identify which hosting provider is the best choice for them. Kevin Ohashi from ReviewSignal has been working with LoadStorm to run a series of load tests on some of the top WordPress hosting providers to determine which is the best for companies who need scalable websites.
Our performance engineers have teamed up with Kevin to analyze the multitude of data and provide this report of the top WordPress hosting providers for web performance. Providers included in this study are: A Small Orange, FlyWheel, GoDaddy, Kinsta, LightningBase, MediaTemple, Nexcess, Pagely, Pantheon, and WebSynthesis. These providers were included in the 2,000 user load test because they didn’t struggle with the first test of 1,000 concurrent users.
This analysis only looks at the final load test of 2,000 concurrent users, but Kevin’s article analyzes the results of both tests and looks at long term up-time reliability. Check out Review Signal’s report of the full study here.
All tests were performed on identical WordPress dummy websites hosted on 10 different hosting services. All sites were tested with the same plugins except in cases where hosts added extra plugins. The websites had identical scripts that included browsing and login. The load tests were run in LoadStorm PRO for 30 minutes with a linear 20 minute ramp up from 500 to 2,000 virtual users and holding at the peak for the for the remaining 10 minutes.
In order to rank the top providers, we have broken our analysis down by the key web performance metrics:
To fairly rank the top providers, we ranked each provider by each performance metric at the 20 minute mark in the test, when all sites were under full load of 2,000 users. For each metric, the providers were ranked (1st through 10th) according to their performance and then a point value was assigned to each. Then we determined our final ranking position based on their total score, the sum of all points from all the metrics.
To view the full test results with interactive graphs in LoadStorm PRO, click on each hosting provider below:
Error rate is probably the most important metric for businesses wanting to be certain that a website won’t crash under high traffic. High error rates mean one thing: Lost customers.
Surprisingly, we had a 7-way tie for first place with 0% error rates. Overall, this speaks volumes to the scalability of all the websites included in the study. Flywheel started to fail at around 1500 concurrent users and began returning 502 errors, which explains its high error rate.
Average Response Time is very significant because it directly affects the user experience and perceived load time. This metric measures the time each request takes “round trip” from the browser sending the request to the server, the server processing the request, and then the response from the server back to the browser. The Average Response Time takes into consideration every round trip request/response cycle for that minute interval and calculates the mathematical mean of all response times.
This metric also measures the same “round trip” that the Average Response Time does, but instead of averaging the time for all requests, Peak Response Time is simply the single longest (slowest) time for a single request.
Average Page Completion Time is a metric that measures the amount of time from the start of the first request to the end of the final request on a page.
In regards to the specific times in this study, the test shows unusually fast Average Page Completion times. After investigating why the pages were loading so quickly, it turns out that some of the pages on the dummy website were very simple with very few requests each. While users with real websites on these providers would expect to see slower average page completion times, the tests are still valid because all providers had the same simple pages.
Throughput is measured by the number of kilobytes per second that is being transferred. This measurement shows how data is flowing back and forth from the server(s). High throughput is a mark of good web performance under load because it shows that there aren’t any bottlenecks blocking and slowing the data transfer. Low throughput, as seen in WebSynthesis, signifies that the server is overwhelmed and is struggling to pass data to and from the server.
Interestingly, GoDaddy pushed triple the amount of data through because their admin screen had more resources being loaded. Which is why the average throughput is so high. Despite the extra data to process, they still had significantly higher average response times than most of the other providers. Anytime a site makes more requests, it slows down performance. Therefore, without so much extra data it is fair to say that GoDaddy could have possibly been faster than all the others.
From the final point tallies, we can see that there are three clear sections.
Top Performers: Pantheon, MediaTemple, GoDaddy, and Kinsta.
Good Performers: Nexcess, LightningBase, A Small Orange, and Pagely.
Fair Performers:: FlyWheel and WebSynthesis.
Overall, most of the providers did surprisingly well under the full load of 2,000 concurrent users. Even though we wanted to rank them in a definitive order, the fact of it is that most providers did not reach failure rates at all in the test. So while we were able to rank them, there were several metrics where the difference between points was negligible (ie: 1 ms average response time difference between GoDaddy and Kinsta) but still calculated in our scores.
Additionally, the test utilized in our report is only part of the full ReviewSignal study. ReviewSignal ran tests at 1,000 users and the providers that crashed were not included in the tests at 2,000. Therefore, all of the providers included in this ranking should be considered great choices for scalable WordPress hosting.
This level of high performance in all 10 providers was unexpected with such a heavy load and we were very impressed by the results.
The post WordPress Hosting Providers Study: Web Performance & Scalability appeared first on LoadStorm.
]]> https://loadstorm.com/2015/01/wordpress-hosting-web-performance/feed/ 25The post The Best and Worst of Cyber Monday Web Performance appeared first on LoadStorm.
]]> Introduction:How the big brand name e-commerce sites handle the heavy traffic on Cyber Monday is always of great interest to our team and our readers. So this year, we decided to run a short experiment on some of the top companies to bring you the best and the worst performers this Cyber Monday.
The 28 companies we chose to test included companies who had painful Cyber Monday crashes in previous years, companies who were running fantastic online deals, and companies that are known to have huge volumes of online holiday shopping traffic.
We ran WebPageTest, an open source performance testing tool, on all 28 companies. All tests were run on Chrome browsers from Dulles, VA at approximately the same times. The first set of tests were run on Wednesday, November 26, 2014 and the second set of tests were run on Cyber Monday, December 1, 2014.
As we categorized the companies based on performance, the most significant factor we considered for this article was time to first byte. Stay tuned for another article where we discuss the speed index and page load times on Cyber Monday.
The reason this article focuses on time to first byte is because it is very significantly tied to perceived load time. If a user waits several seconds and doesn’t see anything loading on the page, he or she is highly likely to abandon the website. However, even if the whole page takes over 10 seconds to load, as long as the user sees progress quickly, he or she can begin looking at the page and is much more likely to stay on the website.
We have ranked the companies into five categories:
Eight companies in our study showed top performance on both Wednesday and Cyber Monday. All of the companies in this group scored an impressive A or B first byte letter score, as assigned by WebPageTest and the highest time to first byte was only 0.315 seconds- Impressive.
Seven companies had moderate performance on both Wednesday and Cyber Monday. These companies were significantly slower than the top performers, but still maintained decent speeds. These companies had scores of B’s or C’s according to WebPageTest. By our assessment, these companies maintained acceptable, but not excellent times.
Five companies in our study had notable performance failures on both Monday and Wednesday. All sites in this group had over a 0.75 second time to first byte, which WebPageTest ranks as a F in their scoring system for time to first byte. Most of these sites had over a full second wait before the first byte was transferred- which is a sign that these sites were overwhelmed by the traffic load.
The level of performance in this category most likely had a significant impact on Thanksgiving and Cyber Monday sales. As we have seen proven time and again by various studies, web performance directly affects conversions. With such significant delays before seeing anything loading on the page, it is very likely that would-be customers left these websites for competitors.
This particular category is not one that we were expecting to see at all. In fact, we initially chose to test on Wednesday as a control group to measure against Cyber Monday. However, a quick poll in our office revealed that most of us had started our online shopping early. It is my theory that these particular companies had extra servers ready for Monday, but did not expect such a heavy load of traffic on Wednesday and were therefore unprepared. It is also possible that when the companies noticed performance failures on Wednesday, they made significant changes over the weekend and then were ready for the Cyber Monday rush.
I’m sure that each of these companies has their own story of WHY their performance was poor on Wednesday and then improved by Monday, but all we can tell you is that it happened.
The previous category was a bit of a surprise to our team. This category, however, was completely expected. As we see every year, there are some companies that just struggle to handle the amount of traffic that hits on Cyber Monday. Check out the differences:
Web performance is a top concern for any e-commerce business because it has been proven time and again to be directly tied to conversions. Just a one second delay in load time has been proven to cause a 7% decrease in conversions, 11% fewer page views, and a 16% decrease in customer satisfaction. With stakes so high, being prepared for the rush of traffic on Cyber Monday is a must for all e-commerce businesses.
Overall, a large portion of the sites we looked at had good web performance on these important days. Even though there are always some websites with poor performance, the general trend is that most websites included in our study were prepared for the rush of traffic.
Feel free to share your Cyber Monday online shopping experiences in the comments! Did you encounter any poor performing websites?
The post The Best and Worst of Cyber Monday Web Performance appeared first on LoadStorm.
]]> https://loadstorm.com/2014/12/best-worst-cyber-monday-web-performance/feed/ 0The post E-Commerce Benchmark – Magento Scalability Results appeared first on LoadStorm.
]]> The Web Performance Lab has been working hard to improve our testing and scoring methodology, and the new benchmark for Magento is here!We had high expectations for Magento because it is such a favorite among web developers globally. Unfortunately, the testing shows that the scalability of an out-of-the-box implementation of Magento was disappointing. Our target objectives for scalability and web performance were not achieved by the load tests, and we saw slower response and more errors at a lower volume than we hoped.
Our Magento e-commerce platform has been installed on an Amazon Elastic Cloud Compute m3.Large instance for a server. Four new scripts to model typical e-commerce user activity have been created:
To be effective in our load testing, we ran a series of preliminary tests. The first of our preliminary tests only utilized one script – new account registration and cart abandonment. This test increased VUsers linearly from 1 to 1000, over the period of an hour. Once the error rates for this test reach an unacceptable level, the test was stopped and the amount of concurrent users at that point are noted. This test allows us to find that rough baseline, while simultaneously registering new user accounts in our e-commerce platform.
The second test we conducted utilized all four scripts, weighting the traffic according to typical e-commerce user activities. This test was run for only 15 minutes, and increased VUsers linearly from 5-500.
Based on the point of failure according to a high error rate, our three scalability tests would be run only to 200 VUsers. The results of the three tests had minimal difference in results, so we chose to analyze the median test for simplicity.
For this series of experiments, we’ve defined three main criteria to determine a point of failure in our load tests on the e-commerce stores. If any of the following criteria are met, then that point in the test is the point of failure and we determine the system to be scalable up to that point:
Performance Metric 1: Error Rates greater than 5 %
Error rates for the Magento platform were nonexistent until the 26 minute mark in the test, where they slowly begin to increase. Error rates didn’t exceed 5 % until 35 minutes into the test, at 145 concurrent VUsers.
We encountered three types of errors during this load test; request read timeouts, internal server errors (500 errors), and forbidden errors (error code 403). An internal server error means that the web site’s server experienced a problem, but could not give specific details on what the exact problem is. Request read timeouts indicate that the server took longer than the time limit LoadStorm’s parameters gave it to respond, so the request was cancelled. After investigating which requests yielded 403 errors, we found that it was the checkout processes in a script that became “forbidden” halfway into the test. Interestingly, the occurrence of these errors appeared almost Gaussian in nature with respect to time, and even dropped back down to 0% at the 53 minute mark.
Performance Metric 2: Average Response Time exceeding 1 second for a request
The average response time for requests in our Magento store start out very good, and remain acceptable up until around 100 concurrent VUsers. It degrades consistently from that point on and reaches just over a second (1.001 seconds) at the 137 VUser mark. If we examine the throughput and requests per second and compare it with average response time, we can see easily graphically see the point in our load test that we begin to reach our scalability limits.
Performance Metric 3: Average Page Completion Time exceeding 10 seconds
The last key metric to analyze is the average page completion time for the home-page for our e-commerce site. Based on the average response time, it was unsurprising that the average page completion time started out very fast, but also reached unacceptable speeds after the 100 VUser point in the test. The average page completion time exceeds 10 seconds, at 134 concurrent VUsers, with an average page completion time of 11.24 seconds.
This means that the average user would have to wait over 10 seconds just for the home page to load, which is completely unacceptable. Studies on the human mind and usability indicate that website response time limits of 10 seconds will keep a user’s attention. A delay greater than 10 seconds affects the user’s flow of thought, and will often make them leave a site immediately. Ideally, what we like to see for page completion time is 5 seconds or less.
Magento had very interesting results, but it’s performance was disappointing because we expected to reach 500 Concurrent Users and higher Throughput. Like we saw for the WooCommerce platform, the point of failure in our load test was determined by average page completion time, a crucial facet of user experience.
Based on our criteria for failure, we’ve determined the system to be scalable up to 134 concurrent VUsers. To score our ecommerce platform, we took four different metrics that reflect volume and weighted them evenly. We compared the actual results with what we considered to be reasonable goals for each metric of scalability. The average of the four different goals give us our overall Scalability Score.
Improving the scalability of any site builds a competitive advantage. Preparing and load testing a site is a way to gain insight on existing bottlenecks to be improved to ensure a pleasurable online shopping experience. It is unrealistic to expect that an e-commerce site will be successful if it is not scalable. It is essential to investigate the performance limitations of any site, and to implement appropriate solutions to improve the overall user experience. The user experience is just as important as the application’s feature set, and it’s vital for web developers to appreciate the scalability and speed of their apps.
Stay tuned for more e-commerce platforms in queue!
The post E-Commerce Benchmark – Magento Scalability Results appeared first on LoadStorm.
]]> https://loadstorm.com/2014/06/magento-scalability-web-performance-results/feed/ 1The post E-Commerce Benchmark – WooCommerce Results appeared first on LoadStorm.
]]> The new e-commerce experiments are here! The goal of these experiments is to expose how well these e-commerce platforms perform in their most basic setup when running on an Amazon m3.large server. To test the scalability of each platform, we will run three hour-long tests to ensure reproducible results. This time around we are utilizing quick, preliminary testing to establish a rough baseline for the amount of users each platform can handle. In addition, the Web Performance Lab has modified the criteria that will be used to determine performance failure.WooCommerce Experiment Methodology
The first platform we will be re-working in this series of experiments was WooCommerce. After initializing the WooCommerce store, a set of four scripts was created to simulate user activity. In our preliminary load test, traffic was ramped up from 1 to 1000 VUsers over the period of an hour, using a single script that completed the new account registration process in the WooCommerce store. Parameterizing this script with user data allowed us to create new accounts that we could use to log in with during later tests, while at the same time obtaining an estimate of how many users the system could handle. Error rates spiked to an unacceptable 14.9% at just under 200 concurrent VUsers, and the test was stopped leaving us with 147 new accounts created in the WooCommerce system.
The second preliminary load test was run using all four scripts and weighting the traffic accordingly to mimic typical user transactions.
Because we already had a rough idea of where the test would fail, we only ran the test for 15 minutes, scaling up linearly to 500 VUsers. This time error rates remained relatively low until around 220 concurrent users, where they spiked to about 12%. Based on these results, we decided to run the three scalability tests scaling linearly from 1 – 250 concurrent VUsers. Each test would hold the peak load concurrency of 250 VUsers for the last 10 minutes of the hour-long test.
Criteria for Failure
The tough question and the crux of why we’re re-doing these experiments really is determining what scalability means.
As performance engineers, we understand that all of the metrics matter and are tightly coupled. However, after a lot of research and discussion, the Web Performance Lab agreed upon a set of 3 main criteria to use to determine a point of failure in our scalability tests, based on if any of the three criteria were met. The following three criteria were used to determine the system’s scalability, considering the impact of the performance scenarios:
Performance error rate greater than 5 %
Average Response Time for requests exceeding 1 second
Average Page Completion Time exceeding 10 seconds
All three tests had very similar tests results, varying less than 1% in the number of overall errors and requests sent between two of the tests. Due to this uniformity, we chose to analyze the median of the three tests. These are the results of the test, based on our new criteria for failure.
Data Analysis & Results
Performance Metric 1: Error Rates greater than 5 %
It wasn’t until the around the 35 minute mark in all three tests, error rates begin to increase, jumping up to 5.05% at 38 minutes in our median test run. The two types of errors we experienced included request read timeouts and internal server errors (500 errors). An internal server error means that the web site’s server experienced a problem, but could not give specific details on what the exact problem is. Request read timeouts indicate that the server took longer than the time limit LoadStorm’s parameters gave it to respond, so the request was cancelled. There were no specific requests in particular yielding these errors, which is good because it means there weren’t any particularly troublesome requests being made in the load test.
Performance Metric 2: Average Response Time exceeding 1 seconds for a request
It’s important to note that this is the average response time for all requests made in a minute interval, not an entire page. Taking a closer look at the performance metrics of all three tests, we can see several key indicators of performance limitations:
In the beginning of the test, the average response time for requests was very low, but it appears to increase in an exponential pattern as the concurrent users increase linearly. We can notice throughput and requests per second scaling proportionally with the linearly increasing VUsers. Interestingly, there appears to be a strong correlation between the point we notice those two metrics plateau, and the point where average response time spikes. This happens just before the 30 minute mark, which is indicative of scalability limits.
Performance Metric 3: Average Page Completion Time exceeding 10 seconds
The last key metric to study is the average page completion time. We chose to focus on the homepage completion for our testing purposes, as it’s the page most commonly hit. We realize we could’ve chosen a smaller time required for a page to complete, but we decided to be generous in our benchmarking. I expected to see page completion time begin to drop off near or slightly before the 38 minute mark, along with throughput and increase in error rate. Surprisingly, our criteria for failure was met at the 29 minute mark, at 147 concurrent VUsers, with average page completion time pushing 15 seconds. This means that the average user would have to wait over 10 seconds just for the home page to load, which is completely unacceptable.
Conclusions – Where does our system fail?
Based on our criteria for failure, we’ve determined the system to be scalable up to 147 concurrent VUsers. Our system’s bottlenecks appear to be on the server side, and our limitation is average page completion speed, a crucial metric of performance scalability.
To score our e-commerce platform, we took four different metrics that reflect volume and weighted them evenly. We compared the actual results with what we considered to be a reasonable goal for each metric for scalability.
Determining a set of criteria to base our scalability on was a challenge. As a young performance engineer, analyzing the results and studying how all of the different pieces correlate has been like solving a puzzle. Based on our original e-commerce experiments, I expected a slightly more scalable system. The metrics we used are absolutely essential as a predictive tool for benchmarking and identifying the real performance limitations in the WooCommerce platform.
What did you think of our experiment? Give us feedback on what you liked and on what you’d like to see done differently below!
The post E-Commerce Benchmark – WooCommerce Results appeared first on LoadStorm.
]]> https://loadstorm.com/2014/05/woocommerce-results/feed/ 0The post Web Performance of Ecommerce Applications appeared first on LoadStorm.
]]> You probably have been wondering why I’ve posted so infrequently over the past year. We have been bombarded with emails and phone calls demanding more blogging that includes my extremely dry, obtuse humor. So, in the interest of global stability and national security, I must acquiesce to the will of the masses.Right. That’s a joke. I’m only slightly more popular than Justin Bieber. If you are a serious tech geek, you’ll need to look up that Bieber reference.
Web performance is why you read our blog, and web performance is my life. I want our blog to contain the most interesting information about load testing, page speed, and application scalability. In order to deliver on that goal, we came up with the concept of using LoadStorm and other tools to gather actual performance data regarding as many web applications as possible.
Thus, the Web Performance Lab was born.
Amazon EC2 General Purpose Types of Server Instances
WPL (I’ll abbreviate because we geeks love acronyms) gives us a playground to satisfy our curiosity. I want to know:
There are hundreds of similar questions that I ponder with Elijah Craig. He and I are very productive in the evenings, and he helps me plan experiments to solve the riddles of web performance that appear to me in visions after a long day of work in the load testing salt mines.
U.S. Online Retail Sales 2012-2017
With over $250 billion of online sales in the U.S. alone during 2013, and with over 15% annual growth, how could we ignore ecommerce? It’s the biggest market possible for our Web Performance Lab to address. The stakes are enormous. My hope is that other people will be as interested as I am.
Cyber Monday 2013 generated $1.7 billion in sales for a single day! What ecommerce applications are generating the most money? I doubt we will ever know, nor will the WPL answer that question. However, some of you reading this blog will want to get your share of that $300 billion this year, and the $340 billion in 2015, so I’m certain that you need to understand which online retail platform is going to perform best. You need to know, right?
Cyber Monday sales data
We have been running some of these experiments during the past few months. Esteban shared some of his results in blog posts earlier. My problem with his work is that some of the conclusions aren’t as solid as I would prefer. I spent some time reviewing his results with him in a conference room recently, and I poked some holes in his logic.
Now don’t get me wrong, I am an Esteban fan. He is a friend and high character guy. That said, we all learn from experiments. We try, we ponder, we learn. That’s how humans gain understanding. As a child you figure out the world by putting your hand on a hot stove. You register that learning experience in your brain, and you don’t do that again. You find out the best ways to accomplish objectives by failing. Just ask Edison. He figured out 1,000 ways how NOT to create a functional lightbulb before he found the correct way. So it is with WPL. We are learning from trying.
Therefore, we are beginning a new series of experiments on ecommerce platforms. We will be publishing the results more quickly and with less filtering. We hope you find it useful and interesting. Please feel free to comment and make suggestions. Also, if you disagree with our statistical approach or calculations, please let us know. Recommendations are also welcome for ways to improve our scientific method employed during the experiments.
The post Web Performance of Ecommerce Applications appeared first on LoadStorm.
]]> https://loadstorm.com/2014/04/web-performance-ecommerce-applications/feed/ 0The post E-Commerce Benchmark – Drupal Commerce appeared first on LoadStorm.
]]>Drupal Commerce is the last stop on our tour of e-commerce platforms. Drupal is a free content management system, and Drupal Commerce is an extension that allows you to build a web store with Drupal. This setup is similar to WooCommerce and Virtuemart, which both rely on their own content management systems.
If you missed our previous posts, we are load testing six different e-commerce platforms and scoring them based on web performance. We continue our e-commerce benchmarking series by giving you a glimpse of out-of-the-box Drupal Commerce and scoring it against the same load test standards as our out-of-the-box Magento store in the first blog post.
To streamline setup, I decided to use a bundle called Commerce Kickstart which contains a single package for both Drupal and Drupal Commerce. So you don’t have to worry about the in-betweens of installing the content management system and extension. How easy is that!
Commerce Kickstart is one of the easiest e-commerce packages to install, much like the Magento and osCommerce packages. There was one blaring difference about Drupal Commerce: there were no example products! This was unexpected because the front page (shown below) clearly shows a fake product slideshow.
Every other store had some sort of example product, even if it was just clipart garden tools. I ended up having to spend time adding sample items. I used an array of image sizes and qualities to mimic the variety in a typical web store.
Strangely enough, the example store came with categories. So I was sure to “stock” the items into those categories. I was ready to load test after an acceptable amount of products were added.
At 3 minutes into each of the three tests performed, we consistently see a spike in performance error rates. A majority of those performance errors are request connection timeout errors which means the load engines are not even able to establish a connection with the target server, Drupal.
Amazon Web Services could be blamed for this. The server is then crashing early on at around 350 VUsers. We expect higher scalability for a large Amazon EC2. So it’s like we’re missing a piece of a larger puzzle that needs investigating.
Drupal Commerce gained a score of 55.7 out of 100. This puts Drupal Commerce in the lower end of the of ranks. The score would be better for Drupal Commerce if the WebPageTest repeat view metrics were not missing. We could not re-test due to time constraints, but we are assuming a value of 60 seconds for all the repeat views. This stops Drupal Commerce from unfairly being ranked as the worst in performance.
At this point, we’ve tested six different open source e-commerce platforms.
Since they are all free and open source, you can give them each a try on your own as well. A deeper analysis of the experiments will be done to make a final ranking of all tested e-commerce platforms. Come back next time and see how each platforms ranks!
This post is part of a series: First – Previous
The post E-Commerce Benchmark – Drupal Commerce appeared first on LoadStorm.
]]> https://loadstorm.com/2014/03/e-commerce-benchmark-drupal-commerce/feed/ 2The post E-Commerce Benchmark – OpenCart appeared first on LoadStorm.
]]>OpenCart is yet another free, open source e-commerce solution for the typical user. When you search for “OpenCart” on Google, you will be surprised by what comes up. The first result is under the domain opencart.us, which is not the one we will test because it is proprietary. The correct OpenCart is the second entry: http://www.opencart.com/
If you missed our previous posts, we are scoring six different e-commerce platforms based on web performance. We continue our e-commerce benchmarking series by giving you a glimpse of OpenCart by scoring it against the same standards as our Magento store in the first blog post.
The installation of OpenCart was more difficult than I anticipated. I ran into issues right at the end of OpenCart setup phase. For a typical PHP site, you get an indication that your installation was successful. I received no indication of a success or failure.
All I was given was the dreaded browser white screen with no PHP errors shown.
After a few more installation attempts and looking through the Apache error logs, I found out what was missing. There was a function in the php-mbstring package that was missing from the server. Once that issue was resolved, I could proceed to checking out the live demo store.
The OpenCart Demo Store
OpenCart’s demo store is easy on the eyes like WooCommerce’s demo store. There is an immediate and modern appeal to the out-of-the-box sample store and theme. Funny, as this also isn’t the first time there’s been a Galaxy S tab in a sample store. I created a customer account to generate the scripts necessary for testing. Afterwards, it was time to load test the store.
Test Results for all OpenCart load tests (click to expand)
The test selected to score OpenCart turned out to be test #1. It had an average response time of 9.6 seconds and an estimated scalability of 912 concurrent VUsers. WebPageTest results for first load time were dispersed as follows:
How did these metrics contribute to OpenCart’s score?
OpenCart is currently second to last in performance overall. The two culprits that attribute to OpenCart receiving such a low score were the following:
I was anticipating for OpenCart to score well because of its design approach, but that was not the case. If you are looking for web performance right out of the box, then OpenCart should not be your choice.
The last testable e-commerce platform is coming up. Check in next week for the final web store, Drupal Commerce, and the web performance winner!
This post is part of a series: First – Previous – Next
The post E-Commerce Benchmark – OpenCart appeared first on LoadStorm.
]]> https://loadstorm.com/2014/03/e-commerce-benchmark-opencart/feed/ 0The post E-Commerce Benchmark – VirtueMart appeared first on LoadStorm.
]]>We’re past the halfway mark for our e-commerce benchmarking experiments where we score a bunch of open source e-commerce platforms for performance. Next up is VirtueMart, which is an e-commerce extension for the Joomla content management system.
The first time through, I had anticipated VirtueMart to be a standalone e-commerce platform. VirtueMart’s official site also doesn’t mention Joomla on the front page, or that VirtueMart is an extension. It’s funny because VirtueMart’s official Twitter page shows that information but their official website doesn’t.
The VirtueMart Sample Data Store
Out of all the web stores I had encountered, VirtueMart was the most difficult to set up. Many times I had to reinstall not only VirtueMart, but Joomla too. I kept receiving the dreaded PHP “white screens of death” when visiting product pages. This was partly attributed to the missing the php-xml package.
After resolving that issue, I was greeted by a store with dead links! I had to manually fix those in the VirtueMart backend.
If you decide to use VirtueMart, I would recommend hiring someone who knows what they’re doing. A graphic designer is not a bad idea either because out-of-the-box VirtueMart is lacking aesthetically. After all, it’s just an extension.
Using the LoadStorm load testing tool, we executed the testing phase on VirtueMart to benchmark it with the other e-commerce platforms. The following charts show our load test results:
VirtueMart Three Load Test Results (click to expand)
The test results are pretty consistent, but will that mean a high score for VirtueMart? We estimate a sustainable scalability up to 1012 concurrent VUsers based on our criteria.
We see performance errors take a sharp increase at 10 minutes. Most are Request Connection timeout errors, meaning some VUsers are having trouble even establishing an HTTP connection to VirtueMart.
Other problems point to the almost immediate jump in peak response time of 15 seconds near the beginning of the test. All it takes is one slow resource, and you get extraordinarily long times.
What will bring down VirtueMart’s score significantly compared to other platforms is its average response time. Unlike peak response time, average response time should definitely be kept low at all times. With an average response time of 10.06 seconds, that’s going to seriously affect VirtueMart’s score.
Performance Errors for VirtueMart Test #3
Before we call it quits, we still have to factor in the WebPageTest results into our scoresheet for a conclusive VirtueMart benchmark. Maybe those results will save this e-commerce platform from a low score. Unfortunately, the WebPageTest load test averages don’t help much.
It appears that VirtueMart has also scored lowest out of all the e-commerce platforms tested so far. With a score of 54.28, it’s not looking good. There are still two remaining e-commerce platforms remaining to test.
I’ve never tried Opencart or Drupal Commerce so all bets are still up in the air.
This post is part of a series: First – Previous – Next
The post E-Commerce Benchmark – VirtueMart appeared first on LoadStorm.
]]> https://loadstorm.com/2014/03/e-commerce-benchmark-virtuemart/feed/ 0The post E-Commerce Benchmark – osCommerce appeared first on LoadStorm.
]]>osCommerce has been around since 2000, making it the oldest of all the e-commerce platforms in this series. You will notice its age when looking at the sample data homepage (The Matrix and Unreal Tournament are hot new items for sale). Despite its age, osCommerce is still in use today; companies design and specialize in it. It’s time to put this seasoned e-commerce veteran to the test. How will it fare against modern supergiants like Magento or the new flashy WooCommerce?
If you missed our previous blog post, we are load testing six different e-commerce platforms and scoring them based on web performance. We continue our e-commerce benchmarking series by giving you a glimpse of out-of-the-box osCommerce by scoring it against the same load test standards as our out-of-the-box Magento store in the first blog post.
osCommerce Sample Homepage: Nostalgic
Like Magento and WooCommerce, osCommerce is simple to set up. The only immediate disadvantage to osCommerce is that its aesthetics are not modernized by default. There are books written on making your osCommerce store better and more appealing.
Regardless of how you decide to style your store, we’re staying on task and testing against the out-of-the-box configuration. We performed three load tests using the LoadStorm load testing tool.
osCommerce Load Test Results (click to expand)
The test we score comes from first test which has the median average response time. Let’s check out the Request Connection Timeout error rates because those have the biggest impact on overall error rates.
Request Connection Timed Out Errors over time
Remember, there are two factors we use for determining a store’s scalability for this series. Those two factors are the following:
Both happen at 21 minutes, which means osCommerce can handle 2111 concurrent VUsers on a large EC2 before it fails. Great! osCommerce is the most scalable store so far; it can handle almost 800 more concurrent VUsers than its closest competitor in our series.
Unlike the other two stores, there are no 500 type errors in the test. My impression is that the older minimal user interface helped to improve the performance of the store, or perhaps their experience in e-commerce just shows.
Even with relatively high scalability, osCommerce still received the lowest score out of the three environments so far. It was WebPageTest load time results that averaged out poorly and brought down this fierce competitor. Notice how the WebPageTest load time averages in this case are higher than 15 seconds. This is because WebPageTest performance tests timeout at 120 seconds (2 minutes).
Ultimately, if you’re looking for good performance, osCommerce is still worth a try. Check out our next e-commerce platform, Virtuemart, in the next part of our series.
This post is part of a series: First – Previous – Next
The post E-Commerce Benchmark – osCommerce appeared first on LoadStorm.
]]> https://loadstorm.com/2014/03/e-commerce-benchmark-oscommerce/feed/ 0The post E-Commerce Benchmark – WooCommerce appeared first on LoadStorm.
]]>A look at Google Trends indicate that WooCommerce has been gaining popularity very quickly over the past three years. This could be due to WordPress already being prolific on the content management side of things. Users who already have a WordPress site can easily implement a web store of their very own. This is in contrast to Magento, which is built for true enterprise-level transactions (even if it’s just Magento Community Edition).
If you missed our previous blog post, we are scoring six different e-commerce platforms based on web performance. We continue our e-commerce benchmarking series by giving you a glimpse of WooCommerce by scoring it against the same standards as our Magento store in the previous blog post.
WooCommerce with Sample Data. It’s so clean!
Novices won’t find WooCommerce difficult to set up since it is just an extension of WordPress. I didn’t break a sweat installing and configuring WooCommerce, even with sample data! I imagine this won’t be the case for all platforms I’ll be testing. Anyway, after setting up my sample WooCommerce store, it was time to begin testing! Again, we are going to run three separate load tests and choose the test with the median average response time.
The Three Load Tests for WooCommerce (click to expand)
The median response time happened to come from the second test in the series. Now let’s check out LoadStorm’s error breakdown page of test #2. I split up the error breakdown into two graphs: 500 type errors and request connection timeout errors. Read request timeout will be omitted for this example.
(Figure 5) Breakdown of 500 Errors throughout Test #2
(Figure 6) Breakdown of Request Connection Timeout Errors for Test #2
Not looking too good. We see 500 Internal Server errors near the beginning of the test (Figure 5). At five minutes and 512 VUsers, 500 errors already reach eight percent! We also see that request connection timeouts are a problem (Figure 6). Request connection timeouts mean we can’t even establish an HTTP connection from our engines to the target server. This doesn’t build a good case for WooCommerce. Error percentage is an important marker of performance.
After selecting test #2, we graded against the scoresheet. The WebPageTests were run at their usual time of 0, 30, and 50 minutes. Notice how WebPageTest scored generously with decent load times. The Achilles heel though is the estimated scalability of WooCommerce at about 412 concurrent VUsers! This is also apparent in the other two load tests. For such a clean and minimal store (with sample data), you would assume that it could handle much more than that. Next time we’re going to inspect osCommerce and see if it has better scalability.
This post is part of a series: First – Next
The post E-Commerce Benchmark – WooCommerce appeared first on LoadStorm.
]]> https://loadstorm.com/2014/03/e-commerce-benchmark-woocommerce/feed/ 0