This post is part of a series: Next

E-commerce environments on the internet come in all shapes, colors and sizes. In our next series of experiments, we delve into the world of open source e-commerce platforms. We are already familiar with one of these: Magento Community Edition. I’ve blogged about Magento extensively in the past, but now it’s time to test it against other free e-commerce platforms. Over six blog posts, we are going to run six different e-commerce platforms and score them on performance. The first one covered will be Magento CE.

Our slightly modified Magento CE store

I chose e-commerce platforms based on two criteria: open source/free software and popularity. I came up with the following list with help from some handy references as to which platforms were most popular:

  • Magento CE
  • WooCommerce
  • osCommerce
  • VirtueMart
  • OpenCart
  • Drupal Commerce

Target Environments

Each e-commerce platform will be installed on top of their own m3.large Elastic Cloud Compute server. Then the testing phase begins. Using the LoadStorm load testing tool, I will run three 5 – 5,000 virtual user (VUser) load tests on each target. In addition, a WebPageTest (WPT) will test the target at the 0, 30 and 55 minute marks of each load test. Using WebPageTest assures 3rd party performance results of the homepage. Finally, the test scripts will be based on the three following activities :

  1. The VUser browses three products (29% of activity)
  2. The VUser adds a product to cart, then abandons (68% of activity)
  3. The VUser goes through purchasing a product (3% of activity)

All platforms are different, so I will run load tests against the default “out-of-the-box” implementation of each store. No extra bells or whistles will be added; just the base install and sample store data.
Once the data is collected, we score all 6 e-commerce platforms based on the scoresheet below.

Scoresheet

In essence, better performance metrics means a higher score. For example, a lower average response time means a higher score in the Average Response Time category; conversely, a higher time means a lower score.

Testing Phase

Here are the Summary results of the three tests ran against the Magento CE target environment.



Magento’s 3 Test Results (click to expand)

Between the three load tests, we selected the load test whose Average Response Time value was the median. For Magento, this was the third load test in the series. For the errors section of the scoresheet, we only consider performance-related errors. This includes 503: Service Unavailable (10.9%), Request Read Timeout (38.6%), and Request Connection Timeouts (20.7%). Uncategorized Errors are also included because the worst case is that they are all performance-related errors.

Magento Test #3 Errors by Type

The total performance error count helps us come up with a percentage for the scoresheet. The final Magento scoring is shown below :

Conclusion

Magento receives a score of 68.68. We will determine where this score stands as we test more e-commerce platforms. As Magento stands so far, we notice slow average response times. WPT First View is reasonably fast compared to the unusual results we got for WPT Repeat View . In terms of performance, Magento does well. However, we still have five other environments to test against. Next on the list is WooCommerce.

This post is part of a series: Next

Similar Posts