Gary Busey's Tuesday Load Testing Notes

I’ve have always liked Gary Busey in movies and on TV. He has starred in everything from Kung Fu to The Buddy Holly Story to the Simpsons to Gunsmoke to Saturday Night Live to Lethal Weapon. His recent appearance on Celebrity Rehab with Dr. Drew showed just how much brain damage he suffered in that motorcycle crash. In some ways, Gary’s life is similar to a good stress test: Ramped up the volume until the system could not respond appropriately to requests made of it.

On to the good stuff about system performance testing.

Performance Metrics Tied to Business Metrics

Mike Kelly at SoftwareSearchQuality.com has an article about developing a working understanding of the business drivers for application performance. He postulates that a skilled performance tester can build better tests through understanding performance goals better through knowing what the business truly needs. He also believes this understanding makes test results more actionable. The key question to ask is, “What do the project stakeholders care about?” Read more about
application lifecycle performance testing and monitoring strategies.

Agile Performance Testing

Maybe the best point in Mike’s post is:
“The best time to be thinking about possible performance issues is when you’re actively designing and writing code. You’re still close to the code. You know what it does. You know how it works. For each day away from the code, the fuzzier it gets in your head. The more developers committing to the codebase, the less you really know about the code you’re interacting with. “

Sound like Agile? Yep, test early and test often. I’ve asked questions at conferences of Agile gurus about performance testing early in the development cycle. All of them acknowledge that frequent perf tests should be run from the beginning of coding. Every build should automatically run a load test. However when you press them to see how often their projects actually follow this model, they eventually admit that the practice doesn’t meet the theory. Rarely, if ever, do teams create perf scripts and execute them until the last 20% of of the project. It’s easy to stand up behind a podium and talk to a PowerPoint slide about what you SHOULD do, but it is not reality that teams put high priority on performance until all the pieces of code are pulled together.

Unit Versus Comprehensive System Tests

A common misconception is that there is one form of performance testing – hit the whole system with volume and see how it responds. That’s one type, but in many cases it is much more effective to test a component on its own. For example, let’s say you are coding a module that inserts data into the database. You should examine the efficiency of the way you coded the process before you test the performance of all the moving pieces that call your module. It will be harder to detect the root cause of a bad SQL statement later when it is hidden in the big pile of other database hits.

Mike makes an argument about using tools that will let you test the speed of units of code. Again, sounds good. I support that idea and recommend it to every developer. You may be able to find some expensive queries or poorly written loop structures. That will be a tremendous help when you are testing the speed and responsiveness of the system to a user where the UI layer is communicating with the application layer which is communicating with the database layer. Identifying resource hogs and slow functions is possible early, and it appears obvious that fixing those in unit testing would be best. Tune the individual parts before assembling them. It has a much higher probability of producing a well performing system.

Realistic Test Scenarios for User Types

Mike talks about “performance profiles”, and I tend to think about the different ways people will use the system. In a typical web app, you might have 70% of your traffic from anonymous lookers viewing your product catalog or reading articles. Maybe 15% of your traffic is going through the shopping experience and buying something (if you are lucky!). Perhaps you have 10% of your traffic downloading white papers or filling out web forms to register for your newsletters. You probably have a few of your internal employees signed in to modify content or update the product catalog. All of those are hypothetical of course, but they represent realistic user patterns. Your test plan should reflect these.

It is also true that during Christmas the percentage of credit card processing could go up by 100% over normal. Your application may have spikes in certain kinds of traffic at end of year. I spoke with PerfDan at the STP conference last year over lunch. He said the TurboTax Online app could see a million users per day during the second week of April. Hmmm, predictable. My suspicion is that most of them are in a panic too!

One of our clients recently released a new book. Realizing that they needed to load test for the anticipated volume, they started building tests with LoadStorm. What they didn’t consider was that most of the traffic would NOT come through their home page to make the purchase. They were sending a large email blast with a link to a specific landing page, and their test wasn’t accounting for that. Thus, their original scenario was essentially useless for predicting site performance. Their home page was horribly slow with hundreds of images and dozens of unnecessary Javascript external libraries. The system stressed out at low volume because the test virtual users bottlenecked at the home page. In reality, they were fine because the landing page was simple and loaded quickly without a bunch of crap. We were happy to coach them to improve the home page, but the real point is that traffic patterns are situational, seasonal, and specific to user types.

Mike makes a good suggestion about finding potential performance problems before they bite you in production. He recommends usage modeling, and I agree. More importantly, you should create multiple test plans to reflect each of your usage models. Test with different scenarios together to reflect what could happen under certain circumstances that are likely to happen – even if your actual server logs don’t show that happening right now.
“Usage modeling isn’t always just about trying to test different worse-case scenarios either. It’s also possibly looking at different ways to utilize excess capacity in the “off season” or testing to see if dynamic provisioning will support certain growth scenarios. You might also try different usage models just so you can build a taxonomy of different load profiles so later on, when you’re monitoring production, you can go back and correlate what you’re seeing in production to different load patterns during your testing phase. Sometimes that can provide insight into potential problems. ‘Oh yea, I remember seeing that before. Last time we saw that we were….'”

Load Testing – Putting the Pieces Together

One of my favorite hobbies is to trace a web transaction all the way through a system. It isn’t always easy. Sometimes I get stuck and need to bring in smarter developers than me – I’m the Omega Geek. It’s a challenge that I recommend for you. WARNING: it is tough to do. Worth it? Yes. Easy? No.

It takes a lot of time and energy investment to walk through the depths of your application code, database queries, stored procedures, UI code, web services, etc. But I tell you that if you ever do this, you will become so enlightened about how all your system components are interacting. You will know what piece of code is calling the next, which methods are heavily used, which queries are unique to a particular button click, and which third party library is referenced yet unused.

It will help you be the best load tester on your team because you will have the best insight into where the bottlenecks can form. You will have a perception into the inner workings that is an outstanding jump start when you begin trying to tune the system performance.

If you work with your whole team (developers, architect, product manager, operations, QA), then everyone learns faster. For example, we have an architect that understands the implications of tweaking settings in a message queuing system that is squarely in my area of ignorance. He has shown me ways to employ the messaging technology to get 1000% throughput increase. I wouldn’t have figured that out so quickly (or never) if I had not involved him.

I like Mike’s comment about the team concept of breaking down the system into one transaction flow:
“It’s almost like practicing a fire drill for a production issue. Teams that can do this analysis well can respond to production issues well. It’s almost like a capstone project – tying together all your models, metrics, and logging into one neat package.”

Here are some good monitoring ideas he presents too:
“Look at the profile of the servers while they are processing that transaction. Understand what the impact might be to the end user. Try to become the world’s foremost expert on what’s going on in the system, for that thin sliver of time, for that small transaction.”

Some Things Haven’t Changed

Sure, web applications are much more complex in 2010 than they were in 1998 when my company started building them. But in some ways, this whole performance testing issue is just like Dr. Hood taught us back in college. He made us follow a single processing thread through the VMS operating system to understand the impact on the VAX and on our application. In many ways it is easier today because the tools for monitoring and testing are better. In other ways it is more challenging because there are so many moving parts to explore.

The bottom line in my not-so-humble opinion is that the best performance testing involves a lot of elbow grease. You must dig in and dig in deeply. It is not a trivial exercise, and you will hit roadblocks along the way that seem insurmountable. However, as long as you set your goal properly to fully understand the inner workings of your application, and if you have the tenacity to push through the complexity of interactions of components, then you will become a world-class performance engineer.

My last bit of advice: Don’t wait to start. Get going on a deep dive now without reading, researching, and studying all the documentation. You can find the answers once you are clear on the questions posed by your system. Dig in and see what wall you hit first. Then start asking people for help. Your colleagues, product vendors, and online forums are great places to help you over that first challenge and on to the next one. Soon you will be amazed at how thoroughly you understand the performance aspects of your application and its environment.