Load Testing Success Criteria – LoadStorm

One of the over-arching objectives of load testing is to measure system capacity. When you are load testing, what are the goals? Generally, everyone wants to know how much load (user traffic) the system can handle without significant performance degradation. What is significant degradation?

Before you begin load testing, I urge you determine what you need from the testing. Is there a service level agreement that your company must fulfill? Are you trying to make a decision about releasing a new web application into production? Does the CTO just want to make the CEO feel good that we’ve done a good job with a new version? Are you creating a performance benchmark? When will we know we are finished with load testing?

Think about the goal of the load test. What are the success criteria and how will we know when the objective is met?

The best answer for all IT questions applies here…”it depends”. Ok, that’s sort of a cop-out. Another good answer that is used by most IT consultants can be used for load testing success criteria…”Whatever will make the customer happy”.

Yes, we need to keep customer satisfaction in mind. Yes, we need to understand who our stakeholders are. Yes, we need to figure out what our boss calls success. Yes, we need to talk to the alpha geek in the basement – he knows everything.

Regardless of how you get there, it is imperative that your team decides upfront what constitutes acceptable performance for your web application.

That said, you should define “good performance” in terms of something measurable. Specifically, we recommend picking metrics that are meaningful to your stakeholders. For instance, 5,000 concurrent users with sub-second average response time and less than 0.3% error rate.

It Must Be Measurable

So the load goals should be clearly quantified, such as 25,000 concurrent users or 400 requests per second. Some people like to measure the amount of bytes flowing in and out of the network – throughput or bandwidth. Everyone wants to understand how the user perceives the system, so a metric like Response Time is paramount. Measuring the errors returned from the server is another key performance indicator. If average response time is 300 ms but you are seeing 500 Internal Server errors, that’s a very bad thing because that’s what the users will see. A quick error is still an error.

Know the targets for load. Know the targets for measuring performance. The objectives must be expressed as something measurable and indisputable.

The Specification

Where do you get the load goals? Often, there are specifications that define the number of users that are expected in a particular period of time, and it is assumed that everyone on the product team is concerned with meeting the expectations in the specification because stakeholders typically view the spec as a “contract”.

Find the Peak

When there is no spec, developers normally have a plan for the number of users they want the app to support without slowing down or generating errors. I recommend looking at the existing server logs to find the peak times of traffic. If you find that 8:00 a.m. EST is 10x the normal traffic, start there. Let’s say that your web application has 1,000 users login between 7:55 and 8:15 a.m., and they hit an average of 5 pages. I would set my load testing goal to be somewhere around 2,500 because I want to know how the system handles 2.5x the regular peak traffic. It may not get there, but as the load tester, I want to know where my app is stress to failure.

Current Traffic Plus Some

If you have no existing server logs because you are developing a brand new application, then talk to the marketing people. I guarantee there is a VP of Marketing (or similar title) in your company that has a number. That person has been assigned an objective for driving traffic to the site through various marketing activities such as advertising campaigns. in fact, many times that site traffic objective will be one of the measurements used during that marketing person’s job evaluation.

Make a Friend in Marketing

Can you think of a better way to endear yourself to that VP than to sit in her/his office and strategize beforehand about handling that volume? They are marketing and probably just assume the site will take the traffic with no problem. They probably won’t sit around and ponder your load testing metrics, but they will definitely have an idea of what they expect from the marketing perspective. Also, they understanding measurements. Their metrics may be about conversion rates, dollar to lead ratios, and views per click. Make no mistake…they understand measuring activity and results.

Whatever you do – find out who is writing the check. Who is the financial sponsor? Where is the money coming from that is paying for the testing? Identify that person and other aspects of the project start to come into focus (surprise). I suggest you interview that person…maybe more than once. Figure out what they need from the site. What do they need from the application? How does it impact their life? If it fails completely, where do they hurt the most? Clearly and precisely understand the pain related to this site’s performance.

It wouldn’t hurt to check with the legal department too.

So by reaching out to them proactively, they will view you as a leader and an ally in the “geek department”. That is a good thing. While you are there, ask them what other expectations they have. Write it all down. You should try to use any of that information in the development of your load testing requirements.

Resource Utilization

Here is where we geeks get excited. Server side metrics seem to be more scientific, and thus, more interesting to programmers and engineers.

Is the database server’s memory more than 80% consumed when the system is supporting 12,000 kilobytes per second of throughput? How much of the web servers’ CPUs are being used when we are handling 350 requests per second? Those are cool questions for us nerds. More importantly, those are metrics you can use in your success criteria. I encourage your team to brainstorm about each major component of your system – with the output of such an exercise to be goals for measurements such as percentages of utilization for memory, CPU, and network bandwidth.

Some people include stress testing when they discuss load testing, and stress testing is about breaking the system. In light of discussing resource utilization, you should consider how any component of the application’s architecture affects the overall performance. What happens if a disk drive dies during heavy web traffic? How do your bottlenecks change when one of the load balancing machines must be rebooted?

When defining success criteria for your load tests be sure to identify constraints in your system, and set some targets for extreme conditions.

Conclusion

Invest the time early to clearly understand what is going to make or break your load testing efforts. Get others involved. Define the measurements for input/load and for output/performance. You need a stake in the ground, even if it is your best guess.

I strongly recommend that you understand what metrics are going to make your boss happy. Don’t overlook the other important stakeholders that reside in other departments like marketing or sales. Having your “ducks in a row” will always be a benefit to you personally as others assess your success.

I have heard it said that response time is a user concern, resource utilization is a system concern, and throughput is a business concern. That sounds clever, but my opinion is that it’s a bit too trite to be useful. However, the point can be applied generally if it makes you realize how important it is to get different perspectives. You as a system architect, programmer, tester, or performance engineer will not have the same point of view for load testing as the Program Manager.

As for the load testing success, the good news is that it is objective. Measurable is unambiguous. Set the goals upfront, and your job is going to be much easier.