A web developer called me the other day and asked, “Is load testing the same as performance testing?”
I pondered for a moment and replied, “Not really.” Hmmmm…now I needed to support my answer. Thus began a hunt for good answers to my client’s question.
Searching Google, gleaning from our interviews, and asking respected colleagues were parts of my pursuit for the truth about load vs. perf. Sorry to abbreviate, but I’ve heard cool engineers shorten it to “perf”, and I desperately want to be cool. (in a geeky way)
The answer isn’t black and white. However, I have heard software testing professionals say load testing is more aligned with black box testing and performance testing with white box testing. Thus, performance involves getting inside the various components of a system.
Strong opinions are expressed by many experienced testers – and the opinions vary. In this post, I will give my perspective and share quotes from people smarter than myself about the topic. Herein lies the results of my hunt.
Personally, I believe load testing is a subset of performance testing. My point of view will be relative to web applications or websites. In the same way web developers usually will view testing differently from QA Managers, non-web software testers may not agree with my narrow definitions of perf vs load.
My Simplified Definitions
Performance Testing = how fast is the system?
Load Testing = how much volume can the system process?
Performance testing seems to me to be much more broad than load testing. Consider:
- A web developer can test the speed at which a page renders in a browser, and that is testing performance. Yet, that test would have nothing to do with load.
- I might analyze the efficiency at which my database processes a single specific SQL query, and the resulting speed of delivery of the records can be the slowest component of the whole page building process. Measuring that speed is about performance, but only one transaction is involved (small load).
- Load testing is usually focused on metrics like requests per second and concurrent users (the cause); whereas performance testing is more concerned with response times (the effect).
How Our Industry Defines Them
Wikipedia has this to say about load testing:
“Load testing is the process of putting demand on a system or device and measuring its response.
When the load placed on the system is raised beyond normal usage patterns, in order to test the system’s response at unusually high or peak loads, it is known as stress testing. The load is usually so great that error conditions are the expected result, although no clear boundary exists when an activity ceases to be a load test and becomes a stress test.
There is little agreement on what the specific goals of load testing are. The term is often used synonymously with software performance testing, reliability testing, and volume testing.”
Note that last sentence – synonymous with performance testing. Yes, I agree. They are often used interchangeably. Hence the need to more clearly define them.
Wikipedia defines performance testing as:
“In the computer industry, software performance testing is used to determine the speed or effectiveness of a computer, network, software program or device. This process can involve quantitative tests done in a lab, such as measuring the response time or the number of MIPS (millions of instructions per second) at which a system functions. Qualitative attributes such as reliability, scalability and interoperability may also be evaluated. Performance testing is often done in conjunction with stress testing.”
Answers.com provides this definition for load testing:
“Load testing is the process of putting demand on a system or device and measuring its response. There is little agreement on what the specific goals of load testing are. The term is often used synonymously with software performance testing, reliability testing, and volume testing.”
Answers.com provides this definition for performance testing:
“In software engineering, performance testing is testing that is performed, from one perspective, to determine how fast some aspect of a system performs under a particular workload. It can also serve to validate and verify other quality attributes of the system, such as scalability, reliability and resource usage.”
SearchSoftwareQuality defines load testing this way:
“Load testing is the process of subjecting a computer, peripheral, server, network or application to a work level approaching the limits of its specifications. Load testing is a part of a more general process known as performance testing.”
SearchSoftwareQuality defines performance testing this way:
“Performance testing is the process of determining the speed or effectiveness of a computer, network, software program or device. Performance testing can also be used as a diagnostic aid in locating communications bottlenecks.”
Friends and Colleagues Speak Out
Now let’s hear from the experts. There have been several web developers and testers that have graced us with their insights via interviews. Those are in our blog, so I plucked out their thoughts on our perf vs. load question.
Roger Campbell says, “I think of load testing being simply testing a system against a simulation of high user load (or other high I/O load of some kind). Performance testing includes other kinds of tests that constrain the system in certain ways or test particular functions for performance with a single user. Tools like YSlow are definitely performance testing tools, but are testing a system’s speed for a single user.”
Ricardo Sueiras says the difference in the two types of testing is, “Simply put (for me) the difference between measuring the end user response and the corresponding impact on infrastructure to support various workloads.”
Lawrence Nuanez says, “My view is that performance testing is the artistic side to this type of testing. You are interested in how the application responds to various levels and types of load. You are tracking response times, throughput, time to load, etc. Load testing is more about running at a high level and seeing how the application responds. You look for performance degradation, dropped sessions, poor garbage collection, etc.”
David Makogon says, “Hmm… I usually talk about these two things as inseparable parts, sort of like Queen’s We Will Rock You and We Are the Champions. I see load testing as the hunt for the “knee in the curve.” That is, start a test, driving minimal transactional traffic to the site, then raise the load level over time, measuring results, until at some point, the response time takes a jump. Performance testing looks at results from a different perspective. I see it as more of a user-experience test. The user will have expectations about response times, graphics downloading, page navigation times, etc.”
Ryan Cramer says, “Without specifically looking for an exact definition, I consider the goals of load testing to be aimed at ensuring availability of service, and performance testing aimed at ensuring quality (or speed) of service.”
James Christie says, “I’d say performance testing is a general term that covers load testing too. Performance testing is a rather vague term covering response times and the application’s ability to cope with heavy loads. Load testing is a more specific term. I think of it as a technique to allow you to carry out performance testing effectively.”
Hunting With Google & Twitter
Googling “performance testing vs load testing” returns several interesting posts about this subject. The #1 result is 5 years old, and is a post by Grig Gheorghiu on his Agile Testing blog. Well actualy 2 blog posts. The second one contains this insight, “I’d like to propose the following question as a litmus test to distinguish between these two types of testing: are you actively profiling your application code and/or monitoring the server(s) running your application? If the answer is yes, then you’re engaged in performance testing. If the answer is no, then what you’re doing is load testing.”
Goranka Bjedov says, “Any time you run a test and you are actually measuring how quickly a system responds to work load, that’s a performance test. So, in a sense, you are answering the question, ‘Given a load X, how fast will the system return the result?’, right? Load test for me means, I’m putting certain load on the system and what I want to find out over a prolonged period of time is how will the system behave. We try to use about 80% of the maximum load the system can handle.”
Kiran Kumar says, “Load testing generally refers to the practice of modeling the expected usage of a software program by simulating multiple users accessing the program concurrently. Performance testing is the process of determining the speed or effectiveness of a computer, network, software program or device.”
J.D. Meier says, “Performance is about response, time lapses, duration … etc. Load testing is about test behavior under normal/peak workload conditions. Load is more about characterizing / simulating your actual workload.”
Vanita Hasija says, “Load testing is the simplest form of performance tests. A load check is usually completed to understand the behavior of the application provided under…the expected number of concurrent users in the application. Performance testing is the check that is completed, with a view to select how rapidly some aspect of a technique performs under a particular workload. It may also serve to validate other quality attributes such as scalability, reliability and resource usage.”
Scott Stirling says, “Any type of testing where the outcome of a test is dependent on a workload (realistic or hyper-realistic) explicitly characterized, simulated and submitted to the SUT. Performance testing proves that the SUT meets (or does not meet) its performance requirements or objectives, specifically those regarding response time, throughput, and utilization under a given (usually, but not necessarily, average or normal) load.”
Kapil Samadhiya says, “Performance Testing is a combination of 3 type of Testing:
- Load Testing
- Stress Testing
- Volume Testing”
Frederic Jean says, “Performance is how fast a single request is processed. Load is about how many requests you can process in an amount of time.” via Tweet 12:29pm, Jan 04
J Wynia says, “Performance testing verifies how well the application performs for a single user. Load testing verifies capacity for users.” via Tweet 5:56pm, Jan 03
Conclusion
Web load testing is:
- similar to, but not synonymous with performance testing
- concerned with the volume of traffic your website (or application) can handle
- not intended to break the system
- viewing the system from the user perspective
- associated with black box testing
Web performance testing is:
- a superset of load testing
- concerned with speed and efficiency of various components of the web application
- useful with only one user and/or one transaction
- viewing the system from the architecture perspective (behind the server side curtain)
- associated with white box testing