The Web Performance Lab has been working hard to improve our testing and scoring methodology, and the new benchmark for Magento is here!
We had high expectations for Magento because it is such a favorite among web developers globally. Unfortunately, the testing shows that the scalability of an out-of-the-box implementation of Magento was disappointing. Our target objectives for scalability and web performance were not achieved by the load tests, and we saw slower response and more errors at a lower volume than we hoped.
Experiment Methodology
Our Magento e-commerce platform has been installed on an Amazon Elastic Cloud Compute m3.Large instance for a server. Four new scripts to model typical e-commerce user activity have been created:
- Browse and Search
- Product search and cart abandonment
- New account registration and cart abandonment
- User login and purchase
To be effective in our load testing, we ran a series of preliminary tests. The first of our preliminary tests only utilized one script – new account registration and cart abandonment. This test increased VUsers linearly from 1 to 1000, over the period of an hour. Once the error rates for this test reach an unacceptable level, the test was stopped and the amount of concurrent users at that point are noted. This test allows us to find that rough baseline, while simultaneously registering new user accounts in our e-commerce platform.
The second test we conducted utilized all four scripts, weighting the traffic according to typical e-commerce user activities. This test was run for only 15 minutes, and increased VUsers linearly from 5-500.
Testing Phase
Based on the point of failure according to a high error rate, our three scalability tests would be run only to 200 VUsers. The results of the three tests had minimal difference in results, so we chose to analyze the median test for simplicity.
Failure Criteria
For this series of experiments, we’ve defined three main criteria to determine a point of failure in our load tests on the e-commerce stores. If any of the following criteria are met, then that point in the test is the point of failure and we determine the system to be scalable up to that point:
- Performance error rate greater than 5 %
- Average Response Time for requests exceeding 1 second
- Average Page Completion Time exceeding 10 seconds
Data Analysis & Results
Performance Metric 1: Error Rates greater than 5 %
Error rates for the Magento platform were nonexistent until the 26 minute mark in the test, where they slowly begin to increase. Error rates didn’t exceed 5 % until 35 minutes into the test, at 145 concurrent VUsers.
We encountered three types of errors during this load test; request read timeouts, internal server errors (500 errors), and forbidden errors (error code 403). An internal server error means that the web site’s server experienced a problem, but could not give specific details on what the exact problem is. Request read timeouts indicate that the server took longer than the time limit LoadStorm’s parameters gave it to respond, so the request was cancelled. After investigating which requests yielded 403 errors, we found that it was the checkout processes in a script that became “forbidden” halfway into the test. Interestingly, the occurrence of these errors appeared almost Gaussian in nature with respect to time, and even dropped back down to 0% at the 53 minute mark.
Performance Metric 2: Average Response Time exceeding 1 second for a request
The average response time for requests in our Magento store start out very good, and remain acceptable up until around 100 concurrent VUsers. It degrades consistently from that point on and reaches just over a second (1.001 seconds) at the 137 VUser mark. If we examine the throughput and requests per second and compare it with average response time, we can see easily graphically see the point in our load test that we begin to reach our scalability limits.
Performance Metric 3: Average Page Completion Time exceeding 10 seconds
The last key metric to analyze is the average page completion time for the home-page for our e-commerce site. Based on the average response time, it was unsurprising that the average page completion time started out very fast, but also reached unacceptable speeds after the 100 VUser point in the test. The average page completion time exceeds 10 seconds, at 134 concurrent VUsers, with an average page completion time of 11.24 seconds.
This means that the average user would have to wait over 10 seconds just for the home page to load, which is completely unacceptable. Studies on the human mind and usability indicate that website response time limits of 10 seconds will keep a user’s attention. A delay greater than 10 seconds affects the user’s flow of thought, and will often make them leave a site immediately. Ideally, what we like to see for page completion time is 5 seconds or less.
Conclusions
Magento had very interesting results, but it’s performance was disappointing because we expected to reach 500 Concurrent Users and higher Throughput. Like we saw for the WooCommerce platform, the point of failure in our load test was determined by average page completion time, a crucial facet of user experience.
Based on our criteria for failure, we’ve determined the system to be scalable up to 134 concurrent VUsers. To score our ecommerce platform, we took four different metrics that reflect volume and weighted them evenly. We compared the actual results with what we considered to be reasonable goals for each metric of scalability. The average of the four different goals give us our overall Scalability Score.
Improving the scalability of any site builds a competitive advantage. Preparing and load testing a site is a way to gain insight on existing bottlenecks to be improved to ensure a pleasurable online shopping experience. It is unrealistic to expect that an e-commerce site will be successful if it is not scalable. It is essential to investigate the performance limitations of any site, and to implement appropriate solutions to improve the overall user experience. The user experience is just as important as the application’s feature set, and it’s vital for web developers to appreciate the scalability and speed of their apps.
Stay tuned for more e-commerce platforms in queue!