In hindsight it’s a little embarrassing to see the amount of misinformation and misinterpretation in this post. Sure, we were competitors fighting for market share, customer attention and sales leads. It’s fair that you’d be taking a highly competitive position against HP. But I had a serious objection to your attacking me as an individual vs. going after HP as a business entity.

As much as I personally wanted to disclose the information about HP’s pricing for the product, I was not able to do so according to the policies set forth by my employer. You were right to call that out, but wrong to make it personal. However, in doing so you demonstrated the internet’s ability to drive market democratization by putting large and small companies on equal footing. In your keenly documented history of our exchange you illustrate how a small company like LoadStorm can wield openness and transparency against the bureaucratic limitations of a large company like HP. That’s a good thing, I think.

So, I’m glad this post is still out there to remind us of where we have been; having harassed one another with limited facts and context-less accusations all for the financial gains of corporate entities. Our exchange was an elemental wake-up call for me to respect my own 20+ years of performance testing and engineering experience. Now in 2012 as an independent consultant (not part of a giant tech company) I dedicate my time to more effective contributions to our industry and discipline.

If all we do is fight battles over tools, we will lose the war on quality.

Mark

First, LoadStorm was designed exclusively for web applications. We never want to do load testing for non-web apps. We are happy to leave that to legacy tools like LR. What do you suppose is the proportion of applications being developed today that are non-web? I would suspect it is small as a percentage. As for inaccessible behind a firewall, yes we do get requests for that; however, they almost always can open a hole in the firewall using an unusual port, the user agent, or white list the IP ranges. No problem here.

If someone wants to pay an enormous amount of money because their network guys can’t (or won’t) tinker for 10 minutes with their firewall during testing, then we happily let that customer pass to HP.

Does anybody have the pricing on the HP hosted service? It does not appear to be published either. Secretive pricing only for “qualified buyers” implies there is something to hide. It does not, in my opinion, represent straightforward open free commerce. Old school. Seems funny that even HP’s customers don’t know if they sell that service. Stealth selling…wonder how that will work out for them?

Gbondy, thank you for your comment. I do appreciate it, and wish you nothing but success in your development endeavors.

Congratulations on 14 years in the industry. I have no problem with joining you in trying to improve the industry.

What is the source of your statement, “Across the industry 29:30 resumes for performance test professionals are outright misrepresentative of the capabilities of the person.”?

I don’t normally receive or view resumes from performance testers. Are there specific skills that are being fraudulently claimed?

Help me understand how I can help with your request.

Scott Price

Your gross assumptions on the cost of 5K virtual users are incorrect, as is your pricing on the controller. Pricing is not linear when you go to ten times the number of users. Moreover you are comparing apples to oranges in many cases. 5000 Virtual users of which protocol? Are you looking at Web? Are you Looking at Oracle? Are you looking at SAP thick Client? Are you looking at Winsock? There are something like 35 different shipped interfaces for LoadRunner ranging from terminals to databases to email solutions to middleware to (yes finally) Web technologies. This is before you add the SDK for roll your own protocols. Let’s not forget the substantial monitoring capability, rivaling that of some systems management solutions, to aid in the collection of information for diagnostic purposes

Again, are you are also looking at the Apples versus Bananas comparison (or perhaps Tangelos) of cost per execution cycle? This is not a unique comparison to LoadStorm vs HP LoadRunner as the comparison breaks for any “performance testing as a service: solution versus an in house owned tool. You could just as easily make this comparison between LoadStorm and Microfocus’ SilkPerformer/QALoad or LoadStorm and Microsoft Visual Studio Teamtest.

If you have these commercial tools you can just as easily place load generators in the cloud if you wish as right next to your app. However, what is very well known in the commercial tools universe is the issue of clock skew inside of virtual machines as well as the VM overload effect on hypervisor distribution of resources on load generators. These issues with clock skew impact response times, collection of metrics on the load generator OS for any per second metrics and the ability to reproduce a test due to cross virtual machine influences on resource allocation under load. These influences are true no matter whether one uses a commercial or open source tool, uses a service or is in house. You are also at the mercy of whatever your VM tells you on the allocation of resources. Hypervisors report bizarre stats back to hosted OS’, such as 230% of CPU, negative RAM allocation, etc…. This is across vendors, none are immune.

I have no problem with your use of Virtual Machines for load generation, so long as you fully disclose all known deviations from best practices and known impacts to timing record based data when test data is delivered. Then the customer can decide if VM load generation is ‘good enough.’ Granted, this is a lot like telling a new mother that her new child is ugly when you tell a developer that their code is slow, so one should expect to have results challenged is known integrity issues are disclosed. Hopefully they do not use the vm skews as an aluminum baseball bat to completely dispatch the results, but I have seen that happen

On April 1st, 2010, I celebrated 14 continuous years of working in the performance testing profession. It’s a profession I love and want to see fortified with high quality performance test professionals, high integrity test results and high value to the stakeholders of our results. For all of us in the profession, we have far larger problems than skewering each other’s tools in many cases. Across the industry 29:30 resumes for performance test professionals are outright misrepresentative of the capabilities of the person. This is across all tool vendors.

In an expanding industry the odds that a new customer will find their way to a fraudulent resource are incredibly high. With a mastery of the language the person is viewed as the expert, but lacking skills they cannot and will not be successful. This impacts LoadStorm as well. Imagine an immature person supplying a faulty load profile, incorrectly configured business processes, an inability to define the correct hosts to monitor. Garbage in: Garbage out. And yes, they will blame the tool and the vendor before they accept blame themselves. They will blame LoadStorm.

I am going to ask you to join many of the other voices in the industry to bring this issue to the attention of customers. The greatest danger to the industry across all tools and vendors resulting in ineffective, high risk, poor performance tests is the incredibly high level of skills fraud in our industry and the appalling low level of architectural expertise as required for the diagnoses of performance related issues.

James Pulley
mailto:[email protected]

Sure, LoadStorm is “cheap” and uses cloud-hosted services to keep the price down. That’s great, but won’t work on non- web apps, or local apps inaccessible behind a firewall, or apps in production without outside internet access, etcetera etcetera.

I think it’s fairer to compare LoadStorm to HP’s LoadRunner-based hosted service, NOT the full boat LoadRunner install. I assume HP still sells that service (Mercury offered it).

Just my observations as a user of LoadRunner, I have no financial or other stake in this argument.

But no argument here that LoadRunner is expensive as (&#$%. It’s definitely a huge investment for a company.