LoadStorm™
On SearchSoftwareQuality, there is a section called Ask the Software Expert where a “guru” answers someone’s question. Scott Barber has a post for Understanding performance, load, and stress testing that I find amusing.
I have customers frequently ask me why LoadStorm has a minimum pause time. They want the load test to click from step to step in less than a second. My reply is that it is unrealistic to have actions of users like that.
People have to think about what they see, read it, make a decision, then click. Each person will respond to the delivered page in a slightly different time frame. Think time is variable because real users have a large deviation in their ability to process information. Some are visual learners, some are analytical, some are contemplative, some are driving personalities,
So when you are load testing, it will be beneficial to have a randomized pause between steps in your scenario. This will realistically represent the length of time from when a person receives a response (page) from your server to the time that person requests a new page.
I used to believe that interpolation for performance metrics was acceptable. However, now I have found that I was wrong. Especially when it comes to think time.
Gartner reports 13.8% increase in mobile device sales in the second quarter of 2010 over 2009 Q2.
Load testers should take note that global access to web sites is becoming more common from smartphones. In fact, smartphone sales are now up 50.5% from the same period in 2009.
One of the over-arching objectives of load testing is to measure system capacity. When you are load testing, what are the goals? Generally, everyone wants to know how much load (user traffic) the system can handle without significant performance degradation. What is significant degradation?
On the Software QA and Testing Resource Center site, there is a post about Estimating Test Duration. Here is a quote for your educational benefit:
I’ve known some web Product Managers that require at least one 72 hour load test on a monthly basis. In fact, a couple are customers of LoadStorm.
The reasoning behind conducting such long tests is that memory leaks can be very difficult to find in a 1 hour test and should be detectable after 72 hours. Comparisons of memory consumption in the first hour to the 72nd hour under the same load may expose any defects in the code that leave memory wasted.
There is a difference between performance testing of a traditional client server application versus a web application.
Web performance testing involves primarily HTTP traffic. There are latencies involved with Internet technologies that you shouldn’t encounter on an internal software implementation.
CTOs Don’t Get It
One of the most disruptive events at organizations is the disconnect between decisions made by managers and the views of those decisions by employees. Does this situation sound familiar?
I found this interesting because it shows how even large companies running on their own “safe” data centers can experience massive performance failure. SaaS and cloud providers may get media attention for outages, but isn’t it somewhat hypocritical of Fortune 5000 CTOs to claim that their internal systems are “safe”? Come on, let’s be real. Systems have been experiencing poor performance for 60+ years, and hardware will continue to fail, and software will always have bugs, and architects will overlook weaknesses, and CEOs will annually cut budgets for performance engineering.
I’ve seen too many people ignore load testing until it was too late. Some don’t test at all. Some treat it with a duct tape approach: Test a little here, test a little there, no planning, no consideration for what is a real-world traffic scenario, and sloppily thrown together at the last minute before going live in production.
In many situations, the correct answer is to ask questions. Software developers, testers, and project managers have a tendency to tell rather than ask because they have so much knowledge and expertise about the subject domain. However, before you begin load testing, you need to know as much as possible to help you create better scenarios and scripts.
I’ve have always liked Gary Busey in movies and on TV. He has starred in everything from Kung Fu to The Buddy Holly Story to the Simpsons to Gunsmoke to Saturday Night Live to Lethal Weapon. His recent appearance on Celebrity Rehab with Dr. Drew showed just how much brain damage he suffered in that motorcycle crash. In some ways, Gary’s life is similar to a good stress test: Ramped up the volume until the system could not respond appropriately to requests made of it.
On to the good stuff about system performance testing.
Performance Metrics Tied to Business Metrics
Mike Kelly at SoftwareSearchQuality.com has an article about developing a working understanding of the business drivers for application performance. He postulates that a skilled performance tester can build better tests through understanding performance goals better through knowing what the business truly needs. He also believes this understanding makes test results more actionable. The key question to ask is, “What do the project stakeholders care about?” Read more about
application lifecycle performance testing and monitoring strategies.
As we begin another week of summer, here are a few helpful links for articles concerning issues of interest to us load and performance testers. We at LoadStorm have been busy the past month with many new customers asking us for help.
In an effort to identify as many performance enhancing suggestions as possible, here are a few more ideas that are specific to Microsoft’s IIS web server. I will try to add to this post whenever I find new IIS performance improvement ideas.
Poor Performance – Your Fault?
Who is to blame for bad application performance? by Alois Reitbauer is an informative look at how developers, system architects, testers, R&D managers, and operations leaders can each play a role in poor performance of software.
While pointing the finger is a common way employees in companies invest their time, rarely does it have much ROI. My experience is that development teams, IT departments, and company executives usually don’t play together very well. They don’t communicate clearly or frequently to each other. It’s only natural because they have their own jobs to do, and their job evaluation (i.e. bonus or raise) isn’t measured by collaboration.
Of course there are hundreds or thousands of posts out on the web about performance testing. I thought I would share 17 good sources focused on web application performance testing. Some of these are lists that lead to other excellent posts.
Ensuring Web Site Performance – Why, What and How to Measure Automated and Accurately
The cloud. It’s exciting. A panacea? Hardly.
On this day in 1994, Israeli Prime Minister Yitzhak Rabin and PLO Chairman Yasser Arafat reached agreement in Cairo on the first stage of Palestinian self-rule. This was the first agreement with the PLO that acknowledged Israel’s right to exist.
On this day in 1469, the Italian philosopher and writer Niccolo Machiavelli is born. A lifelong patriot and diehard proponent of a unified Italy, Machiavelli became one of the fathers of modern political theory. The term “Machiavellian” is used to describe an action undertaken for gain without regard for right or wrong.
Apparently, not everyone agrees with my buddy Steve Jobs. That seems obvious to me because I have a large number of friends and colleagues that are very much in the Microsoft collective. However, I was quite surprised to find that a Netscape hero would be taking a firm public stand against Steve’s State of Web Development and defend IE so vehemently.