There was a time a few years ago that I needed to hire a Senior Performance Engineer consultant to help me set up load generation servers. I also needed help figuring out all the pieces for getting my performance testing environment setup.

There are so many moving parts, I was overwhelmed. Not just servers, but software tools as well. Buying an enterprise tool from a vendor was impossible - their sales reps were quoting me six figure solutions. Picking an open source tool was hard because there are so many and each has strong advocates, but all required quite a bit of work to get installed and configured to run.

I thought I was the only one struggling. Shouldn't this be easier? Can't I just find a simpler way?

Well, this blog post focuses on one piece of the load testing environment puzzle. I was reading a question thread on LinkedIn in the Load & Performance Testing group. The title (question) is:

What could be the causes if hits/sec keeps going down while the response time and resource utilization are consistent?

The question was asked by Jun Zhuang, Sr. Performance Test Engineer at MedPlus. There were several answers offered by different performance engineers and experts. All of them were applying skill, experience, and logic to the problem in an attempt to help Jun. What I found most interesting is how one consistent idea kept being presented:

You're load generators aren't working correctly.

Ramanjaneyulu Narra, Performance Engineer at Axeda Corporation, suggested two possible causes for the hits per second decrease: 1) Caching or 2) Load generators "are being dropped". Not sure how a load generator gets dropped, but he was definitely on the right path with the idea of something was wrong in the load generation part of the testing tool.

Navaneetha Krishnan, Performance Test Lead at Compuware, responded, "Looks like the load generators could be a bottleneck here."

Later, Jun posted this, "it seems there are problems with the new load generators we setup, after a little while they just stopped sending in requests. But we have yet to pinpoint the cause."

Dave Collier-Brown, Senior Staff Engineer at TrekLogic (a Sun Partner), subsequently posted his thoughts. He states, "Yes, problems with load generators are endemic to the industry, plus the failures are often "silent", and aren't visible to the tester, or subtle enough that they can easily be missed. Bummer! "

Dave also mentions that when the response time metrics don't show you the famous hockey stick curve, "it's a good indication of something broken in the experiment, oft-times in the load generator. "

After Jun did some additional analysis and investigation, he concluded, "The problem still turned out to be with the VMs being used as load generators. All the load testing VMs reside in the same corporate VM farm as other VMs and the person who created those load testing VMs did not give them enough priority or shares of resources."

Chuck Masters, Performance Testing Coordinator at Allegis Group, followed up with this insight, "I personally have not used VM boxes as Load Generators and even though HP says it is "supported" I have spoken to many a tester that says don't use load generators on VM. It doesn't work like a traditional stand alone load generator."

Donald Foss, Global Load Testing Services at Keynote, contributed this, "I don't generally recommend testing using VM machines because they can introduce artificial delays. They can be manageable if you are experienced with it, but that experience can be painful to get."

My thinking from digesting this discussion is that load generation can be challenging at best - a show stopper at worst.

Buying or acquiring hosted server machines is one obstacle. They cost money. If you make them dedicated to load testing, then they will probably be sitting idle much of their useful lifetime. That's wasted money.

Another challenge is dealing with the software complexity. Operating systems, web servers (including VM), and load testing tools are three primary components of making the load generators work. All of these must be configured properly to get the proper result. Obviously, Jun was struggling with achieving the desired goal of creating load in a manner that would produce valid performance test metrics.

How about this common question that is so fundamental that it is sometimes overlooked:

How many load generators do I need?

Ben Simo addresses this question in his blog post by that title. It depends. Again, it is another aspect of complexity that consumes time and energy before a load test can ever be created.

Ben says:

The real question being asked is: How much load can I put on a load generator without impacting performance?

Isn't this one of the most common questions that load testing attempts to answer? Performance testers, of all people, should understand that there is no single formula for determining how much load you can place on a load generator system.

That "system" includes computers, software on those computers, and the network between the load generation computers and the system under test. To determine the requirements for this system we need to monitor that entire system. We need to monitor CPU usage. We need to monitor memory usage. We need to monitor whatever other system resources the script and test tool may impact. We need to monitor bandwidth usage. We need to monitor at the start of a test and at the end of a test. We need to monitor and load test the load generation environment just as much as we need to monitor and test the system under test.

Is Jun's Situation Unusual?

Or is this a common problem shared among anyone trying to performance test their applications?

From the responses, I conclude that load generators are a challenging piece of technology. These are professionals that work for large corporations with significant specialization in their development teams. These are performance engineers and people with "Senior" in their titles. These are NOT lightweight, rookie IT weenies that took a PHP course. Some of them even work for large performance consultancies (regarded as the best in the world).

For world-class load testing pros like these guys to say that, "problems with load generators are endemic to the industry", that clearly tells me that it's not easy. It's unequivocably a red flag for a time trap.

Load generators can be a sink hole for your energy, money, and time.

Wouldn't you rather invest your time finding bottlenecks and tuning your application? Isn't the goal to make good software and make a high quality user experience? Is it a worthwhile use of company money to pay you to work on intricacies of a technology that won't enhance your career or add value your customers?

Here is Jun's Conclusion

"This experience promoted me to think load testing via cloud. Cloud is VMs anyway, we can't even control the VMs within the walls of our own companies, how can we be guaranteed the resources when we need them from a cloud?"

If you find yourself coming to the same conclusion as Jun, my suggestion is to try a free cloud-based load testing tool. Find out for yourself if that approach will solve the "endemic problem in our industry".

Obviously, LoadStorm is one of those solutions. We can abstract all of that load generation complexity so that you can focus on getting the real job done. We aren't the only cloud load testing tool, and you may want to try others.

Try a free LoadStorm account. No VMs to worry about. No servers to install or configure. The load generation just works - almost instantly. Storm your site without the hassle. We guarantee you'll like the results.

You can invest your time making your boss happy and improving your performance tuning skills. Or you can waste time building the infrastructure from the ground up.