Many CIOs do not embrace cloud computing. They have pressures from stakeholders to make system service levels “five nines”, and clouds don’t offer those SLAs. The trade-off is between highly redundant and available systems at a high cost to maintain compared to much lower cost without the guarantee of high availability. Cloud server platforms are relatively very inexpensive, but it isn’t perfect.
I have seen the same IT infrastructure needs cost 100x in a private data center versus a cloud. That can be a huge amount of money. And the larger the dollar amount, the more religiously it is justified by IT leaders to have their own private data center.
CIO online magazine has an article that touts the benefits of using the cloud for development and testing environments.
I agree, and that is why we built a load testing tool utilizing the Amazon AWS cloud. It is cheaper. It is flexible. It is massively scalable.
However, I think the author of the article has a perspective that is better suited for writing website content rather than performance engineering. I agree with his statement, “Many of the most important bugs only surface under high system load;”. Additionally, he is correct in saying bugs found in production are more expensive to fix than bugs found early in development. But he never clearly comes out and states that you can replicate the production environment in the cloud. He implies that putting the dev system in the cloud and hammering it with load will let you find the performance-related bugs better than in a traditional internal data center lab.
I like his thinking, however I must say he has reached an invalid conclusion. It is impossible to replicate your production environment that is not in the cloud with a dev/test system that is in the cloud. Let me repeat that:
Load testing against your app running in a cloud will not replicate the production app performance
You run your app in your private data center with thousands of environment variables, network components, server configurations, and infrastructure settings. Each one of these moving parts will affect the results of your load testing. The mathematical probability of your app running the same in two different environments is effectively zero. Making the case for cloud computing is a good thing. However, the jump in logic is irresponsible that equates testing your app running on a cloud with testing your app in your data center.
Load testing should be conducted against your production system in off-hours. The load generators should be in the cloud, not your application. If you cannot live with “three nines” accuracy for an SLA, then I would question why you can live with an unpredictable statistical difference in load testing results. Those performance metrics gathered during testing of your application running in a non-production environment could be orders of magnitude inaccurate. You cannot rely on those measurements.
I like CIO magazine, and I like this writer. I just had to say something because good people could be lead to believe a great fallacy in performance testing.
One small variable can potentially affect performance measurements by 100% or more.
Skeptical? Just try turning on/off caching. Or try running tests against the same environment except that the version of Apache or IIS is different. You will be surprised at the performance deltas. Isn’t it easy to overlook the size of MySQL buffers or perhaps PHP options or other configuration selections? All of those variables in the application engineering equation can and will affect your outcomes.