Forgive me for stating the obvious, but web applications are a critical part of global business in 2011. I see no alternative other than more dependence by companies everywhere on web software and Internet infrastructure. In my opinion, all business trend data predicts greater overall web usage, more complex application architectures, and tremendous spikes in extreme traffic volumes.

Critical Applications, Yet They Aren't Getting the Investment Needed

ComputerWorld last week made a definitive statement regarding the critical nature of web applications:

Those who are unprepared are vulnerable to service outages, customer dissatisfaction and trading losses - and often when it hurts the most. Successful businesses understand the need to assure service and application availability if they want to retain customers, deliver excellent service and take maximum advantage of the opportunity their market offers.
This is not a theoretical problem - just look at the recent challenges for the London 2012 Olympics andTicketmaster. Just when everyone wants to do business with you, you’re not available.

The London Olympics site was overwhelmed by high demand for tickets and many buyers received the message, “We are experiencing high demand. You will be automatically directed to the page requested as soon as it becomes available. Thank you for your patience.”

That's a failure even if the representatives of the site said it had not crashed. Performance failure...pure and simple for the whole world to see.

Examples of performance failure like this seem to occur weekly, if not daily, somewhere in the global business universe of websites.

Transformative Moment? When Global Retailers Fail!

Recently Target.com crashed under extreme user volume. They cut a deal with a designer line of knitware (Missoni) and promoted a special sale on the morning before products were sold in stores. By 8:00 a.m. EDT, the site was crashing. The Boston Globe went so far as to say:

”...the Missoni mess could be a transformative moment in the relatively brief history of e-commerce. Retail analysts say it shows that even though online shopping has made major strides since Victoria Secret’s website famously faltered during a 1999 webcast, companies still may not always have the technological muscle to meet consumer demand for such frenzied promotions.”

Jason Buksh is a Technical Project Manager and Performance Consultant in London, England. Jason has extensive experience with performance testing at many companies including HSBC and Siemens. He is skilled with tools such as Rational, Grinder, and Performance Studio. His certifications include LoadRunner.

We appreciate his time to share some good thoughts with us about a topic that gets us excited. Here is his interview with us.

What is your technical background?

I learned to program when I was 13 - it was a vic20, I then swiftly moved onto 6502 for the BBC micro. University studying computer science was an obvious and easy progression for me. First job was writing rendering engines (C++) for virtual reality simulators. I would describe myself as a techie at heart - I’m genuinely interested in how things work. I think having a strong and long background in IT enables me to grasp new concepts easily - which is great when I have to go into different companies and need to understand their systems quickly. I've a 2:1 in Computer Science, ISEB Practitioner and SCRUM Master Certified.

Do you consider yourself more of a software developer or QA professional?

We should get out of the habit of separating them so readily. I feel strongly that every software developer should QA their work. Its not good enough to code and then relinquish QA to another team. Its lazy, increases delivery time, wastes effort and increases cost. Everyone should be a QA professional within their own field. I think a large dedicated QA team is a good measure to the inefficiency of an IT project. I’m going to write a post on this very topic.

How much involvement do you have with load and performance testing?

My career is built on it. I’ve performance tested many mission critical and highly transactional systems. Companies like Expedia have extremely large volumes of traffic, and the performance of such a system is paramount. My experience at global financial institutions has taught me a great deal about trading platforms and the importance of milliseconds in response time.

What is the biggest change you have witnessed in the way people conduct load testing?

There is a quiet move away from Loadrunner and its going to become an avalanche. It's been underdeveloped and overpriced for a long time.

Load and performance testing your website is important. Tuning it can have more than a 1,000% return on investment.

Some of the case studies referenced below show us that we can improve revenue 219% just by improving performance our site. Other data confirms that the average business loses $4,100 per hour when their site slows down under load. An outage costs $21,000 per hour on average. Retailers can lose $100,000 per hour.

It's possible to have a successful ad campaign or a wonderful Slashdot day that your site can't handle - and that can send
46.9% of your traffic to your competitors. Or worse, cost you 150,000 customers.

Take Web Performance Seriously

At the beginning of 2009, Denny's made a bold marketing move. At a time when many Americans were out of work, and those with jobs were struggling to make ends meet, the restaurant chain cut an ad offering every American a free breakfast. The ad only aired once, during a little event called the Super Bowl.

And that's when Denny's troubles began.

Within minutes of the ad airing, customers who attempted to access Denny's Web site to get their free meal coupons found that they couldn't get through. The company planned the commercial and the $3 million ad buy perfectly. What it didn't plan for was the ensuing deluge of traffic, which Internet marketing expert Rob Kmiec estimates represented anywhere between a 434% and 1,700% spike in the daily traffic to dennys.com. Had the company planned for the additional attention and invested in a Content Delivery Network (CDN) or cloud network to handle the ensuing load, Kmiec argues, it would have reaped nearly a 1,000% return on that investment. Instead, the company lost the opportunity to serve an additional 153,300 customers.

Stress tests are good for finding out how many people can be on your website at the same time. A good test plan for an e-tailing site includes scenarios that represent people browsing your product pages and searching for specific items. It should also have realistic traffic for buyers going through the shopping cart experience and purchasing products with a credit card.

I had a call yesterday from a customer that wanted to stress test his e-commerce site with about 100,000 concurrent users. He explained that their marketing department is expecting significant growth of their sales because of a combination of the economic outlook, increased advertising, and a cool new unique channel technology that is going into production.

Thanks to some data from Gallup, Experian, and Pew Research, Marketing Charts produced some great graphs that show how the U.S. economic financial growth indices are improving in the past year or so. Consumers are spending more online with e-tailing sites, perhaps because they are more confident that their financial situation is improving.

Online Spending is Healthy

In January, 58% of survey respondents said 2011 will be better than 2010, 20% said 2011 will be worse, and 21% said it will stay the same. As the mood has improved, so has online spending which hit a record of $43.4 billion in the fourth quarter of 2010. That's up from $32.1 billion in
Q3 2010, and it is the fifth consecutive quarter of positive year-over-year growth and second quarter of double-digit growth rates in the past year. So, online spending is one of the most vigorous parts of our economy right now. Is your site ready for growth?

I read a good blog post about load and stress testing that brought out some good points, and I agree so much that this post will provide my view of the subject.

In the load-testing.org article, it states that old thinking dictates buying your own dedicated servers for generating load and licensing legacy software to run test scripts. It's expensive, slow, cumbersome, inefficient, wasteful. Not only are tools like LoadRunner expensive, they require considerable training and experience in staffing.

Another key point in the article is about infrastructure costs:

"The advent of Cloud Computing offers a new model - one that eliminates the burdens of capital investment and load testing configuration. Cloud Computing uses the federated power of a large number of servers to provide on-demand processing power to online applications, which rent only as much processing power from the cloud as they need.

This post is part 1 and will cover my thoughts on the way cloud computing has changed load testing forever. Tomorrow will have part 2 and will focus on the impact of Agile on stress and performance testing.

Old School Paradigms of Load Testing

My experience in programming over the past 30 years (I'm not really that old) has shown that just about everything related to application performance has changed significantly. Even the terminology. A new paradigm has arrived, and I'm embracing it.

Joe Emison is the VP of Technology at BuildFax. He is also the Chief Systems Architect at BUILDERadius. A world-class multi-tasker, Joe lives in Ashville, North Carolina.

Joe has experience with load and performance testing, and we are grateful he invested his time to share some good thoughts with us about a topic we are excited about.

On an interesting note, Joe is also a JD graduate from Yale Law School. How many alpha geeks do you know that went to law school? It's a rare breed that only the folks at LoadStorm can hunt down. ;-)

Let's start the interview.

Please share one tip or best practice that is important to you regarding performance (or load) testing or engineering?

Before you start, you need to understand how everything works. Down to this level of detail:

• How many queries does your application make to the database for different activities?
• How many connections should your application make to the database server?
• How much memory should your application be using?

Otherwise, you won't know how to use the results of load testing.

A successful website fails quickly. Successful with high volumes of visitors. Failure in performance under load. Lots of marketing money and energy wasted.

Web Performance (fast response) is important - 46% of people abandon sites because of slow load times.

If load testing and performance engineering was easy, my boss wouldn't have hired a geek like me. A swimsuit model would have gotten my job.

Google found that an extra 500ms in latency cost them 20% of their search traffic.

If you give someone a web application, you will frustrate them for a day; if you teach them how to code a web application, you will frustrate them for a lifetime.

A load tester learns more from one HTTP status codes of 500 than from 1 billion status codes of 200.

I'll probably take some heat for this post. Most professional testers get deep into the details of arcane aspects of the science of load testing because that is their job. It's important to them, and they have studied it for years, and they are immersed in it. I understand. It helps to differentiate them from other testers that are not as knowledgeable; thus, it is a competitive advantage to incorporate as many functions of performance as possible. Consultants certainly need to show off their advanced skills gained from decades of load testing so that the customer can be assured of getting good value. Again, I understand. I respect these highly trained testers and engineers.

That said, I'm of the opinion that many times the professionals spend 80% of their time building load test scenarios that only have a 20% impact on the performance metrics. Implementing these nuances into scripts and testing plans makes the project more thorough, but from a business ROI perspective, gettting that extra 20% of accuracy is NOT worth the 80% of effort. The following paragraphs discuss some examples of considerations that I recommend you ignore when building your test scenarios.

As we begin a new year, hoping it will be usher in global economic recovery, I would like to share what I can find about the collective forecasting in our industry. In my search, I found several predictions about stress testing the financial systems. Hmmm...interesting, but not helpful.

One really cool article on Business Insider showed the predictions of visionaries from 1931 of their view of 80 years in the future. William Ogburn had the best view of our high-tech world:

Technological progress, with its exponential law of increase, holds the key to the future. Labor displacement will proceed even to automatic factories. The magic of remote control will be commonplace. Humanity’s most versatile servant will be the electron tube.

Fascinating! Bravo Mr. Ogburn. Although it wasn't the electron tube, you have the right idea. Wonder what will replace silicon chips in the next 80 years?

Mobile Performance Testing Predictions for 2011

Joshua Bixby, the President of web performance company Strangeloop, puts forth these prognostications in his article entitled 2011 Web performance predictions for the mobile industry:

1. Companies will generate at least 15% of Web sales via their social presence and mobile applications.
2. Android will become the No. 1 mobile platform, surpassing the iPhone in terms of units and usage.
3. Retailers will realize that mobile shoppers have a goal-driven “hunter” mentality.
4. As a result of No. 3, mobile Web performance will become as important as desktop Web performance.

Recently, Tom Cagley of Software Process and Measurement Cast interviewed me concerning load testing. Tom's blogs and podcasts are focused on interviews, essays, facts and tips about process improvement and measurement in the Information Technology arena. INTERVIEW IS HERE

Tom is Vice President, Director of Process Improvement and Measurement at David Consulting Group, and the author of the hit book Mastering Software Project Management: Best Practices, Tools and Techniques. The book has received all 5-star ratings on Amazon.com.

Our load testing interview not only talks about web performance and stress testing, but it touches on several ancillary subjects of interest to Tom from his perspective with Agile software development. I have worked with LEAN manufacturing process, and Tom applies many of the best practices from this successful discipline in his approach to software development. One mantra of LEAN is, "You can't control what you can't measure." Tom has a keen emphasis on measurement of any metric that can help a team deliver better software.

In fact, he has written several blog posts about Seven Deadly Sins of Measurement Programs that could be quite beneficial to developers studying load testing because measurement of metrics is critical to all web system performance.

I found an interesting blog post today through a tweet that HootSuite found referencing LoadStorm. It is cool to see someone blogging about using our load testing tool.

Today we had a customer push the limits for a high measurement of requests per second in our performance testing tool. Their test hit a peak of nearly 1,800 requests per second! Wow, that pretty good.

We are still making some improvements to our our own system bottlenecks in AWS, but our team is obviously making some great progress. Yeah, I know. The irony is not lost on me. ;-)

I ran into a Tweet today that said, "Is #loadtesting to know *where* it beaks, not *if*. That's it does isn't the issue, that I know where and roughly when is #load #testing". The tweet caught my eye because of the hash tag for loadtesting.

The implication, as I interpreted the 140 character bit of wisdom, is that load testing answers the questions:

1. How much load breaks a system?
2. Where does the system break under load?

Identity Access Management (IAM) is a very complex aspect of IT. Finding someone that really understands it is a challenge. I have found one of those guys.

Corbin Links is an expert in the implementation of various forms of access management systems - commonly called Single Sign On by those of us that need to simplify. He is the author of a trilogy of books entitled "IAM Success Tips Volume 1-3".

Corbin has gotten involved with load testing because many times his projects for large enterprises requires his team to ensure performance of IAM. We at LoadStorm are fortunate in having Corbin as a user of our load testing tool, and he has been a delightful customer. Recently, he asked if I would come on his podcast to share some thoughts about performance testing relative to IAM.

I have customers frequently ask me why LoadStorm has a minimum pause time. They want the load test to click from step to step in less than a second. My reply is that it is unrealistic to have actions of users like that.

People have to think about what they see, read it, make a decision, then click. Each person will respond to the delivered page in a slightly different time frame. Think time is variable because real users have a large deviation in their ability to process information. Some are visual learners, some are analytical, some are contemplative, some are driving personalities,

So when you are load testing, it will be beneficial to have a randomized pause between steps in your scenario. This will realistically represent the length of time from when a person receives a response (page) from your server to the time that person requests a new page.

I used to believe that interpolation for performance metrics was acceptable. However, now I have found that I was wrong. Especially when it comes to think time.