I read a good blog post about load and stress testing that brought out some good points, and I agree so much that this post will provide my view of the subject.
In the load-testing.org article, it states that old thinking dictates buying your own dedicated servers for generating load and licensing legacy software to run test scripts. It’s expensive, slow, cumbersome, inefficient, wasteful. Not only are tools like LoadRunner expensive, they require considerable training and experience in staffing.
Another key point in the article is about infrastructure costs:
“The advent of Cloud Computing offers a new model – one that eliminates the burdens of capital investment and load testing configuration. Cloud Computing uses the federated power of a large number of servers to provide on-demand processing power to online applications, which rent only as much processing power from the cloud as they need.
This post is part 1 and will cover my thoughts on the way cloud computing has changed load testing forever. Tomorrow will have part 2 and will focus on the impact of Agile on stress and performance testing.
Old School Paradigms of Load Testing
My experience in programming over the past 30 years (I’m not really that old) has shown that just about everything related to application performance has changed significantly. Even the terminology. A new paradigm has arrived, and I’m embracing it.
Back at EDS in the 1980’s we called it volume testing, not load and stress testing. It took months to properly setup the tools and the target environment to simulate a few hundred concurrent users against a mainframe/COBOL system. We felt like we were cutting edge because most people didn’t even know what we were doing, and when you tried to explain it to them, they shook their heads and mumbled something like, “Your wasting good development time”.
In the 1990’s most people were concerned with performance of applications running on their PCs. Load and stress testing had more to do with comparing the cool new 386-33mhz desktop from CompuAdd (remember them?) to the machine on which we were running our FoxPro-based claims processing production system. Every bit of performance was coveted, but we really didn’t have a complex application architecture to tune. Improving SQL query efficiency was the #2 priority in our performance engineering – second only to the hardware itself. Ok, I guess some things haven’t changed because most managers now still throw bigger servers at performance problems, and database queries are still a common problem.
Elastic Cloud is a Fundamental Change
Most things have changed however. The paradigm around load and stress testing has definitely and irrevocably shifted. LoadStorm is a beneficiary of this huge adjustment in the web development world. In 2007, the lightbulb went off for our CTO, Roger Campbell, while reading some Alpha Geek magazine on his vacation. The article explained how data centers were starting to offer “elastic computing” that would allow us web product managers to forego buying lots of expensive hardware – we could start renting it by the hour whenever we needed it. Hallelujah! That was a breakthrough in efficiency.
Smaller companies can now be competitive in a quickly growing web application world that could see spikes in traffic of 10,000s of users within minutes because they can load test their sites. Essentially, without the new paradigm of performance testing a team would simply wait for the customers to put load on the system. The old paradigm was “publish and pray”.
All we needed was to invest in writing software to take advantage of that elastic nature of computing. Hence, the birth of LoadStorm. It required learning on our part to build a web system from the ground up on the new Amazon EC2 API. The complexity slowed us a bit at first, but soon we were rocking the platform. It felt like the turbochargers kicked in after a few months of coding and experimenting. The results were exciting.
We are proud to be a part of saving web development companies $100,000+ dollars on their load testing. Your testimonials and high-five emails have made our life rewarding over the past 3 years. All of this progress in cost-effectiveness around our load testing tool is due to a new paradigm – and not just the elastic cloud.
Summary
Tomorrow’s post (Part 2) will address how the old paradigm of stress testing right before launch has changed. Agile development encourages testing early and testing often, so web apps are getting more attention for performance in every release. The new way of thinking is to run load tests over and over – even after every nightly build to see how overall performance was affected by code changes.
I’ll wrap up by sharing another good quote from the load-testing.org article:
“Critical changes – such as improvements to a processor-intensive business algorithm, or database enhancements made by a DBA – can be tested and re-tested on the spot, greatly accelerating the software development process.”