Cyber Monday Performance Evaluations – LoadStorm

From Amazon to Argos, online retailers are experiencing more traffic than ever this holiday season. This Cyber Monday reached a record high of $2.68 billion! Competition is fierce, and in this game, seconds = $$$.

This year, we selected 29 major e-commerce sites and used LoadStorm to run several tests to compare their performance on the Wednesday before Thanksgiving with Cyber Monday. We created scripts for each site to model typical e-commerce user activity. Each script would hit the homepage, search for a product, add a product to the cart, and then visit the cart. Then we ran our performance tests for 10 minutes at a time, scaling from one to ten virtual users (vusers).

Here’s what we found:

Average Page Completion Time remained relatively reasonable.

Out of the 29 companies, 7 slowed down:

Walmart’s average page completion time increased from 2.9 seconds to 7.8 seconds. That’s huge! Amazon, on the other hand, remained consistent, with an average page completion time of just 1.6 seconds on both days. Average Page Completion Time remained nearly the same for the majority of the sites.

Six companies sped up:

Performance Error Rates increased.

Our preliminary load tests revealed zero performance errors across the board, with one exception. Monday, however, was a different story, as we saw five different companies experience performance errors. This included request read timeouts, request connection timeouts, and even some 503 (service unavailable) errors.

The exception to the increase in performance error rates was Best Buy. Interestingly, we saw Best Buy experience seven request read timeout errors on Wednesday (on product pages and search results), but none on Monday. This seems to corroborate the fact that they became overwhelmed with traffic over the weekend, but they appeared to have recovered gracefully by Cyber Monday.

Peak Page Completion Times increased.

Every site we tested experienced high peak completion times. Some of the best performers with the lowest peak page completion time on both Wednesday and Cyber Monday included Toys “R” Us, Brookstone (one of our customer’s – yay!), Ikea, and Amazon, with peak page completion time only deviating 7% from their mean.

Web Page Test Results

Web page tests were on each company’s home page were performed during testing on Wednesday and Monday as well. Surprisingly, the overall trend was a decrease in page load times. The average page load time from our web page tests decreased from 9.3 seconds to 5.6 seconds. We’re impressed!

Overall Performance

Nobody crashed while we were running our performance tests. Most sites appeared to perform reasonably well, but knowing that just one second delay could cost Amazon over $1.6 billion in sales over the course of a year those few errors matter immensely. Just 250 milliseconds, either slower or faster, is the magic number for competitive advantage on the web. So while none of the sites crashed entirely, whether or not they beat out their competition is another story.

Please note that none of the companies involved were contacted nor paid for participation in our experiments. These were just for fun. Here’s the complete list of the companies we tested:

• Walmart
• Macy’s
• JCPenny
• Amazon
• Target
• Victoria’s Secret
• Finish Line
• Best Buy
• Toys “R” Us
• Urban Outfitters
• Crate and Barrel
• Barnes and Noble
• Nordstrom
• Ann Taylor
• Motorola

• Newegg
• PC Mall
• Avon
• Footlocker
• Brookstone
• Kohls
• Radioshack
• Ebay
• Argos
• Ikea
• Home Depot
• Office Depot
• Staples
• Sam’s Club