stress testing

The team over at Promet Source is sharing their expertise in a webinar showing you how to properly stress test your Drupal site. Andy Kucharski has asked me to join him in sharing best practices with anyone needing to understand the load testing process.

Register for the webinar

Andy is a frequent speaker at Drupal camps and conferences all over the country. He is recognized as one of the top minds in the world when it comes to architecting an effective Drupal implementation. Andy’s reputation recently has skyrocketed as the foremost performance specialist on Drupal technology.

Real Drupal Performance Data from Promet Load Tests

One of the coolest parts of the webinar will be Andy’s data that compares the performance of Drupal hosted on Amazon versus Rackspace. His team has done some real load testing with Drupal identical applications installed on different environments. There are statistics for AWS EC2 Small and Medium instances, and there are stats for Rackspace Cloud 2G and 4G instances.

Since most of our readers are software/web developers with global reach, we want to share with you info about an upcoming international conference where you can learn cool new coding techniques and hang out with some rock stars in the developer industry. LoadStorm is a proud sponsor of this software development conference and is offering 5,000 free virtual users in our stress testing tool to attendees!

Stress tests are good for finding out how many people can be on your website at the same time. A good test plan for an e-tailing site includes scenarios that represent people browsing your product pages and searching for specific items. It should also have realistic traffic for buyers going through the shopping cart experience and purchasing products with a credit card.

I had a call yesterday from a customer that wanted to stress test his e-commerce site with about 100,000 concurrent users. He explained that their marketing department is expecting significant growth of their sales because of a combination of the economic outlook, increased advertising, and a cool new unique channel technology that is going into production.

Thanks to some data from Gallup, Experian, and Pew Research, Marketing Charts produced some great graphs that show how the U.S. economic financial growth indices are improving in the past year or so. Consumers are spending more online with e-tailing sites, perhaps because they are more confident that their financial situation is improving.

Online Spending is Healthy

In January, 58% of survey respondents said 2011 will be better than 2010, 20% said 2011 will be worse, and 21% said it will stay the same. As the mood has improved, so has online spending which hit a record of $43.4 billion in the fourth quarter of 2010. That’s up from $32.1 billion in
Q3 2010, and it is the fifth consecutive quarter of positive year-over-year growth and second quarter of double-digit growth rates in the past year. So, online spending is one of the most vigorous parts of our economy right now. Is your site ready for growth?

I read a good blog post about load and stress testing that brought out some good points, and I agree so much that this post will provide my view of the subject.

In the load-testing.org article, it states that old thinking dictates buying your own dedicated servers for generating load and licensing legacy software to run test scripts. It’s expensive, slow, cumbersome, inefficient, wasteful. Not only are tools like LoadRunner expensive, they require considerable training and experience in staffing.

Another key point in the article is about infrastructure costs:

“The advent of Cloud Computing offers a new model – one that eliminates the burdens of capital investment and load testing configuration. Cloud Computing uses the federated power of a large number of servers to provide on-demand processing power to online applications, which rent only as much processing power from the cloud as they need.

This post is part 1 and will cover my thoughts on the way cloud computing has changed load testing forever. Tomorrow will have part 2 and will focus on the impact of Agile on stress and performance testing.

Old School Paradigms of Load Testing

My experience in programming over the past 30 years (I’m not really that old) has shown that just about everything related to application performance has changed significantly. Even the terminology. A new paradigm has arrived, and I’m embracing it.

Here are a few good insights about software performance from Robert Read in his eBook entitled, How to be a Programmer: A Short, Comprehensive, and Personal Summary. Robert dedicated the book to his colleagues at Hire.com.

My favorite parts are listed here as excerpts and included below in the original context:

• Bottlenecks in performance canl be an example of counting cows by counting legs and dividing by four instead of counting heads.
• The purpose of stress testing is to figure out where the wall is, and then figure out how to move the wall further out.
• If the wall is too close to satisfy your needs, figure out which resource is the bottleneck (there is usually a dominant one.) Is it memory, processor, I/O, network bandwidth, or data contention?
• Performance is a part of usability.
• Most software can be made (with relatively little effort) 10 to 100 times faster than they are at the time they are first released.
• If a well-isolated algorithm that uses a slightly fancy algorithm can decrease hardware cost or increase performance by a factor of two across an entire system, then it would be criminal not to consider it.
• A plan for stress testing should be developed early in the project, because it often helps to clarify exactly what is expected. Is two seconds for a web page request a miserable failure or a smashing success? Is 500 concurrent users enough?
• I’ve made errors such as failing to provide a relational database system with a proper index on a column I look up a lot, which probably made it at least 20 times slower.
• Other examples include doing unnecessary I/O in inner loops, leaving in debugging statements that are no longer needed, and unnecessary memory allocation.
• Stress testing is fun.
• Who has particular knowledge about a component also constantly changes and can have an order of magnitude effect on performance.
• Finding the expensive I/O and the expensive 10% of the code is a good first step
• There is not much sense in optimizing a function that accounts for only 1% of the computation time.
• Each change brings a test burden with it, so it is much better to have a few big changes.

I ran into a Tweet today that said, “Is #loadtesting to know *where* it beaks, not *if*. That’s it does isn’t the issue, that I know where and roughly when is #load #testing”. The tweet caught my eye because of the hash tag for loadtesting.

The implication, as I interpreted the 140 character bit of wisdom, is that load testing answers the questions:

1. How much load breaks a system?
2. Where does the system break under load?

Today I read an article by Omer Brandis on IT Toolbox about SAP stress testing versus performance testing. Maybe his perspective differs from mine because he is not focused on web applications. I just haven’t seen other technologists use the definitions and goals of the two types of testing like Omer uses them.

I’m reading the classic book The Art of Software Testing by Glenford Myers. While it may be old (copyright 1979), it is still full of golden nuggets. Myers uses real examples from the ancient days of software development to demonstrate the practical side of testing.

One of my favorites is his definition of stress testing.

“Stress testing is the process of evaluating failure points in a web application by introducing adverse conditions into the runtime environment.”