I ran into a Tweet today that said, “Is #loadtesting to know *where* it beaks, not *if*. That’s it does isn’t the issue, that I know where and roughly when is #load #testing”. The tweet caught my eye because of the hash tag for loadtesting.
The implication, as I interpreted the 140 character bit of wisdom, is that load testing answers the questions:
- How much load breaks a system?
- Where does the system break under load?
Very good observation. I re-tweeted and replied. He then wrote a full blog post to explain his thoughts called Tested to Destruction.
I like Jeremy’s post. However, if I just agree with it, there wouldn’t be much reason for my blog post, would there? 😉
Thus, I continue.
There are so many different semantics games that can be played with the definitions of load testing, stress testing, and performance testing. I have written a few articles about the subject, and one in particular has gotten several rave responses including a couple of new customers. Forgive the self-serving plug, but I recommend you check out Load Testing vs. Performance Testing
I see stress testing as designed to break the system – which may or may not involve heavy load. It can be unplugging a drive in a RAID array to see how a system holds up under the stress of a failed hardware component.
Load testing is applying traffic and measuring what happens. You can run a successful load test at a volume of concurrent users that does not crash the system. For example, your marketing department may say their goal is to support 1,000 concurrent users. If your test shows an acceptable response time at that volume, then that’s load testing at it’s best.
Performance testing can include both stress and load testing, but it can encompass more. As an example, tools like YSlow can measure the performance of an individual page. It’s valuable to see how each resource is delivered and the time breakdown on requests/responses with rendering time factored in. That’s certainly not “load” or “stress” because it is analysis of a single page, but it is testing performance.
Anyway, my favorite part of Jeremy’s blog is this paragraph:
“Having a 48G web server with quad hex cores pointed at a single HD machine with a 5400rpm ATA drive might look good when you’re running your PHP with no dB access, though hit it with 5000 requests that use table locks and see what happens.”
Very true. Unfortunately, it is too common for web developers build a load test scenario with all anonymous users. It is much more beneficial to have authenticated users because the web application must hit the database. Personalized pages won’t use nearly as many resources from cache, such as the HTML that is build just for John Doe. You MUST load test with as much database
involvement as possible because most performance bottlenecks that occur when a web application is under load are related to the database. Expensive queries, locked tables, and repetitive writes are all going to hurt performance.
Now, here is one area where I point out a difference in my thinking as opposed to Jeremy’s. He includes determining “WHERE” the system breaks in load testing. I would call that performance engineering. The results of load testing will be metrics like response times, requests per second, error rates, throughput, etc. If I want to find out that my system is breaking in the load balancing configuration, then I probably need to do a little engineering to figure that out. Finding the location of poor performance is challenging. It requires gurus like Jeremy, who is a LAMP expert, that understand the whole technology stack and architecture. The interaction of moving parts can often mask the root cause of a bottleneck.
Thus, I believe Jeremy takes for granted how smart and experienced he is. Many load testers can understand the metrics and can tell you how many concurrent users can handle before response times get excessive or 500 server errors begin appearing. Not all load testers can analyze system log files which include some cryptic clues to a problem with performance. That’s a job for the Alpha Geeks. Admittedly, I’m the Omega Geek on our team. (whimper)
Wait…if he means “WHERE” is the volume that breaks the system, then ok, I agree. Thanks Jeremy – you are probably smarter than me.