Perry Reed has worked in performance testing for eight years, and he has extensive experience with load testing for large software applications. Perry has applied his skills at such companies as Publix Supermarkets and Home Shopping Network. Currently, he works for the largest company in the world – HSBC (according to this page on LinkedIn).
We appreciate Perry sharing some of his insights and best practices with us. This performance testing interview helps us performance testers add to our knowledge of the craft.
Please share one tip or best practice that is important to you regarding performance (or load) testing or engineering?
Plan your testing carefully! Get agreement from all stakeholders on what you are going to test, how you are going to test it, and how you’ll decide if the test has passed. Never run a test without a well-defined and agreed to plan.
What do you believe to be the most critical elements of web application testing? Why?
Know your users! Specifically, know how they’re using your application and in what numbers. I can’t tell you how many applications I’m asked to test and the application owners have no idea how many users they have or what transactions are used most often. Good production application monitoring tools are not that expensive and should be part of every web app.
Once we know how your application is used and what sort of volume you get, we can design a performance test to simulate that use, including peak volumes or occasional stress situations. We can also use your production information to determine our test success criteria. I’ve been asked to run tests for application owners who will say, “Just run the test and we’ll see what the numbers are. We don’t know what level of performance is considered OK.”
I also like to focus on the “high volume / high impact” transactions only. Don’t get bogged down into testing every transaction of an application, many of which rarely get used and have a minimal potential impact on performance. Stick to those transactions that are done all day long in high volume and those that have potential to impact performance (like complex searches or report building).
What is your technical background?
I first learned to program a computer using a Commodore PET and later an Atari 800. I wrote more worthless programs back then than almost anyone I knew. Professionally, I graduated college with a degree in Business Administration / Information systems and then started out as a COBOL programmer. This was many, many years ago. I moved into project management after a few years and later into QA and then QA management. More recently I have returned to project management, but with performance testing as my area of specialty. I enjoy mixing my expertise in performance testing with the more “standard” aspects of project management. I currently work as a Technical Project Manager for HSBC.
Do you consider yourself more of a software developer or QA professional?
Really, I consider myself a project manager who happens to know a lot about QA and specifically performance testing. In my current job, I split my time between traditional project management tasks and providing technical consulting on performance testing.
How do you see development or testing evolving over the past few years? the next few years?
On the corporate side I haven’t seen too many drastic changes recently, although I’ve been glad to notice a greater interest in performance testing by management and business teams. Out in the “real world” I see a lot more web applications that are aggregating content from other web applications, like social networking sites. I’d love to see how some of those get tested.
Is there anything commonly overlooked in web application testing?
Sometimes we’ll get asked to test an application that already has an inherent performance problem in one or more transactions. These transactions don’t run well with even one user, and there is no way they’ll pass a performance test. I advise my customers to conduct some application profiling and dig into those transactions and fix them well before we attempt any large-scale performance testing.
How much involvement do you have with load and performance testing?
It is my primary focus. I enjoy it quite a bit more than functional testing, although I do think it is a lot tougher in many ways. Since many in management don’t really understand it, it requires a lot more explaining to justify larger (and more expensive) test environment requirements, and to agree to a test plan. I have found that without a well-documented and inclusive test plan, agreed to by ALL relevant stakeholders, you’ll get a lot of potshots taken after the testing, from executives wondering why you tested in a particular way or why you excluded certain transactions, and so on.
What would you say is the difference between load testing and performance testing?
My personal definition of load testing is that it is a type of performance test in which you simulate peak load volumes on the application. This differentiates the load testing from capacity or scalability tests (both kinds of stress tests) or endurance (soak) tests.
What do you think is the most important aspect of load testing?
Certainly making sure you have an accurate and realistic simulation of production load is key. I have done some large projects where an application is tested over several weeks, with many tweaks and configuration tuning done to resolve identified issues, only to find that when the application goes live, it fails with performance issues because the user activity was different than expected.
How do you determine the load to apply to the target app during load testing?
We usually target the peak production volume, (I like to call it, “The busiest hour of your busiest day”) and base our tests on that. Often we may have to scale the volume for the performance environment; for example, when we’re using two web servers compared to four in production. The stress tests will also be based on the peak load, adding 10-25% additional volume, usually.
Do you prefer using requests per second (RPS), concurrent users, or some other metric to define load?
We use both, generally. RPS is probably the most important metric, but the number of concurrent users can have an impact on memory on the web server, especially when a lot of view state or session information is stored. In those cases, the number of concurrent users tested should match production as accurately as possible. In other cases, we do occasionally increase volume by increasing the rate of requests rather than adding additional users and this is especially helpful when you have a limit to the number of virtual users available (usually by licensing issues).
Do you feel like performance testing is an accepted critical part of the development life cycle?
I think at my current company it has become so in the last couple of years. That hasn’t always been the case, so I’m glad to see it.
How important is web performance to your clients?
It is critically important, especially for external customer-facing applications. Not only must you guarantee a good experience for users, but you must ensure that your system has the capacity to handle peak loads.
What resources do you typically remove/restrict from a system to perform a stress test?
Whenever possible, we like to “stub” any back-end systems that are called from the web application to isolate the web app during testing. However, we often will also tests an application “end-to-end” to stress both the front-end web app as well as back-end components, like MQ messaging, and back-end Unix or mainframe applications that are providing data to the front-end. This can introduce issues when trying to stress the back-end since typically a mainframe application can handle far more load than the front-end web app. We will occasionally stress the back-end components directly by firing MQ messages that simulate the front-end application.
If you could make a career from one of your favorite hobbies, what would it be?
I’m a private pilot and if I could fly for money, I sure would!
HSBC Holdings plc is a public limited company incorporated in England and Wales, headquartered in London. It is the world’s largest company and the world’s largest banking group (Forbes list, 2008). HSBC was named the world’s most valuable banking brand by The Banker magazine (2008).
HSBC was established in Hong Kong and Shanghai in 1865 to finance the growing trade between China and Europe by Scotsman Thomas Sutherland, who wanted a bank operating on “sound Scottish banking principles”.