David Makogon tweeted something interesting a couple of weeks ago. It caught my attention and seemed to be very accurate, so I re-tweeted it. His post was:
“Agreed. LoadRunner is powerful, but steep learning curve+cost. 80/20 rule: do you “really” need it?”
So after I re-tweeted David’s post, I got a response from @mtomlins that said:
“@loadstorm If you need speed, confidence, accuracy, scalability and reliability…then you need #LoadRunner and it’s worth every cent”
That struck me as a haughty reply from someone that must think very highly of their own opinion. Load testing tools may have different functionality and some are better than others, but LR isn’t the only solution.
I have no reason to argue with David’s comment about learning curve and cost. It’s no secret that load testers, QA professionals, and performance engineers all over the world know that LR is the most expensive solution available. Wonder why @mtomlins wanted to argue about it?
Load testing can be successfully conducted by open source tools, cloud tools, desktop tools, and commercial tools like LR. For many of us, LR is not worth every cent. It’s beyond our budget.
Here is the thread of Twitter exchanges that ensued (I’m @loadstorm):
loadstorm: @mtomlins so are you saying there is only one solution to get speed, confidence, accuracy, scalability, and reliability in an application?
mtomlins: @loadstorm There are perf testing tools that do a few things very well. I would argue that #LoadRunner does very many things very well.
loadstorm: @mtomlins Agreed it does many things very well. @dmakogon said so. His point was it’s very expensive. What’s cost for one 5,000 user test?
mtomlins: Expensive?…depends on the comparison. 3 years of LoadStorm costs about $35k, which isn’t pocket change for most testers.
loadstorm: @mtomlins How much would 5,000 concurrent users cost for 3 years if I used LoadRunner?
So far I haven’t received an answer from him. Notice how he deflected the question. I was truly curious about this guy at this point. Why would he go out of his way to try to make LoadStorm sound expensive? What motivation would he have to even be so silly as compare LoadStorm vs. LoadRunner on price?! You gotta be kidding me!
Ah…..he is the product manager for LoadRunner. He’s the HP guy that must defend his baby. That’s probably why he avoided telling me what LR costs for 5,000 users. LoadRunner pricing must be better than LoadStorm? Let’s analyze that theory.
Well, I went to HP’s site and found their Product Catalog and searched on “LoadRunner”. It returned:
0 products shown below matched your search on “loadrunner”
I tried searching the whole HP site for pricing, but couldn’t find any. Lots of white papers and brochure descriptions, but no pricing. You have to call their sales reps. Guess they don’t want to scare people.
Then I tried Google Answers. It shows the following as LoadRunner pricing back in 2005:
$22,000 Controller $55,000 500 virtual users ———– $77,000 Subtotal $15,400 20% cost for support mandatory for first year ———–
$92,400 First year software & support fees
$15,400 Second and following year support fees
Considering that @mtomlins chose to quote a price for 5,000 virtual users at unlimited test runs, I thought I would try to calculate an apples to apples price for LoadRunner. It seems to me that LR would charge this:
$572,000 License fee for controller and 5k vusers $343,200 Support for 3 years ($114,400 annually) ————
$915,200 Total HP software fees
$915,200 for LoadRunner 3 Year Software Fees
That’s just HP’s charges for the 3 years. We need to determine the full cost that compares fairly with LoadStorm, therefore we must add in costs for hardware/hosting LR. But alas, I searched in vain for a tangible answer to how many vusers can be run on a certain server configuration. I had to use what I could find and make some assumptions for this calculation.
Based on an Amazon EC2 large instance (7.5 GB memory, 2-core, 64-bit platform) generating 500 virtual users, servers for controller/monitoring/analysis, and bandwidth, hosting costs should be about $50,000 for the 3 years.
Hmmmm….David also mentioned the learning curve.
I have read blog posts from performance testing experts that claim it takes years to really understand and become proficient in LoadRunner. It is classic legacy enterprise software. I’ve worked in legacy enterprise software for probably 20+ years of my career, and it is clear that adding features is the only way to keep selling new versions. Build it bigger. Make that feature into a parameter with 5 options. Yep, been there…done that.
There is no question in my mind that LoadRunner is a feature-rich software product. I have heard that it is very flexible and has complex functionality to allow scripting at several levels. It has so many options and ways to build tests that it requires special training classes from HP.
There are 11 courses listed on HP Training Page. For example, LoadRunner Essentials is $3750 for a 5-day course.
Therefore, by my best estimate, it probably takes a million dollars and several months of learning to use LoadRunner. That’s what David was talking about. The 80/20 rule would say that you can probably conduct your performance testing and satisfy about 80% of your requirements with 20% of the cost.
If my math is correct, then $35,000 is only 3.5% of $1,000,000. Let’s review the comparison that @mtomlins is making:
| LoadStorm | LoadRunner |
|---|---|
| 5,000 vusers | 5,000 vusers |
| 3 yr Total Cost | 3 yr Total Cost |
| $35,000 | $1,000,000 |
| Pocket change | Ridiculous! |
Ok, maybe David was wrong with the 80/20 rule application. @mtomlins, it’s actually WAY more expensive than that.