Load Testing in the Cloud

Interview with Head Geek Andrew Lombardi

Andrew LombardiAndrew Lombardi is a Java Software Engineer and Head Geek of Mystic Coders, LLC in Santa Ana, California. His blog provides insight into many aspects of technology.

Our thanks to Andrew for taking the time to share some information with us.

Stress Testing Vs. Performance Testing - Different Opinions

Today I read an article by Omer Brandis on IT Toolbox about SAP stress testing versus performance testing. Maybe his perspective differs from mine because he is not focused on web applications. I just haven't seen other technologists use the definitions and goals of the two types of testing like Omer uses them.

Stress Testing: T-Rex the Guy in the Outhouse

I'm reading the classic book The Art of Software Testing by Glenford Myers. While it may be old (copyright 1979), it is still full of golden nuggets. Myers uses real examples from the ancient days of software development to demonstrate the practical side of testing.

One of my favorites is his definition of stress testing.

"Stress testing is the process of evaluating failure points in a web application by introducing adverse conditions into the runtime environment."

Wikipedia & Performance Testing Definitions

wiki and load testing /></p>
<h3>Happy Birthday Wikipedia!</h3>
<p>As a child, my mother made an investment in a set of World Book Encyclopedias for my education.  Those books came in handy so many times - especially on the night before a paper was due. If the subject was worth a teacher assigning homework about it, then it was in the encyclopedia.</p>
<p>Well, today I would like to recognize the 9th birthday of Wikipedia.  With a tag line of

I thought it would be fun to see how Wikipedia defines some of the key terms we deal with in web application development and testing.

Load Testing vs. Performance Testing

A web developer called me the other day and asked, "Is load testing the same as performance testing?"

I pondered for a moment and replied, "Not really." Hmmmm...now I needed to support my answer. Thus began a hunt for good answers to my client's question.

Searching Google, gleaning from our interviews, and asking respected colleagues were parts of my pursuit for the truth about load vs. perf. Sorry to abbreviate, but I've heard cool engineers shorten it to "perf", and I desperately want to be cool. (in a geeky way)

Performance Testing New Year's Resolutions 2010

It's that time of year. Set some new goals. Make some changes that will have 2010 run smoother than last year and accomplish more than ever before.

Two key parts to this post:

  1. Performance Testing Frequency
  2. 8 Principles to Succeeding in Your New Year’s Resolution

Performance Testing Tip of the Week

In the spirit of New Year's resolutions, I'm going to try to provide as many tips of the week as I can. There are weeks when I might get too busy, but I'll sincerely try to post a small hint or trick that will be helpful to web developers. Tips may be about load testing, performance testing, performance tuning, stress testing, or anything related to web application development.

Performance Reference Point

Happy New Year

As I sit here on this Monday morning and try to figure out what I should do first, it hits me that today is the beginning of an exciting year. 2009 will probably be remembered by most people as the big global economic recession. For me, 2009 will be significant because of the launch of LoadStorm.

Performance Tuning = 10% Profit Increase

Your website is a little slow - so what? Well, it is probably costing you money. I have been researching published facts about web performance because we are always trying to understand our industry better. This post should help you realize that improving your web application performance can directly impact your bottom line by 10% or more. Don't believe me? Read on...

Response Over 6 Seconds is the Top of Your List?!

I read an article today on the E-commerce Times site Web Performance Metrics That Matter. It was the first result in a Google search on "web performance metrics", and the title sounded like a perfect hit. My intent was to see what other people think are the best performance metrics.

Here is a paragraph that really surprised me:

Load Testing for the Slashdot Effect

What is the Slashdot Effect?

The Slashdot effect occurs when a smaller website is flooded with visitors, usually as a result of being linked to from a high profile website. The name comes from the Linux and Open Source blog http://slashdot.org, a site which handles 80 million page views a month. Many home servers or low bandwidth sites that post content interesting to one of these larger blogs have experienced significant downtime when the traffic from Slashdot suddenly migrates to their own servers. Performance engineering for the Slashdot effect is essentially engineering for a worst case scenario - but that worst-case involves more traffic than developers could ever have imagined!

Performance Testing Kills GM?

There are some excellent stats gathered about performance testing at this resource site.

I especially like the one that states 5 second response time is the cutoff point for the business doing well. If the app takes longer than 5 sec to respond, the company employees start to get frustrated. It makes sense to me that productivity will go down. That not only leads to unhappy employees, it invariably leads to lost customers and lost profits.

Performance Testing Interview with Ricardo Sueiras

performance testing with RicardoRicardo Sueiras is a self-professed "IT professional and geek" (found that on his blog. His formal title is J2EE Systems Architect at PricewaterhouseCoopers LLP in Canterbury, United Kingdom.

Regression Load Testing

NetworkWorld's Senior Editor Denise Dubie provides her views on What is considered good application performance?

The article essentially concludes that whatever the end user can tolerate is probably adequate performance. I was hoping for more technical content and details of studies or something like that.

Don't Get Caught With Your Britches Down (hat tip to Capt Kirk in Wrath of Khan)

CIO Promotes Cloud for Load Testing

Many CIOs do not embrace cloud computing. They have pressures from stakeholders to make system service levels "five nines", and clouds don't offer those SLAs. The trade-off is between highly redundant and available systems at a high cost to maintain compared to much lower cost without the guarantee of high availability. Cloud server platforms are relatively very inexpensive, but it isn't perfect.

Load Testing on Oct 26

On this day in 1992, a lack of load testing and poor software quality assurance practices possibly caused as many as 45 deaths in London.

Load Testing with Cockroaches

Ok, this comes under the heading of "weird stuff" and it is a stretch to call it load testing. My thinking is that this guy is running a test to see how many Madagascar hissing cockroaches fit into his mouth. I don't think it is stress testing because he isn't trying to exceed the capacity of his mouth. In stress testing, he would be trying to break the system; however, in this instance of the test, he is merely trying to break a record (11) previously set by a guy in Kentucky.

Estimating Web Application Load

Queueing Theory and The Science of Waiting

The more you learn in the field of web development (and technology in general), the more that the scenarios in 'Num3ers' seem possible. 'Rubbish!' I thought when I first saw the show. 'Maths can't predict that! There are too many variables! There's human error to account for! You'd have to be omniscient to calculate that!'. Well, queueing theory is making me start to think about whether there is a literal way to eat my words. Web developers use it to do things like calculate the probability that a given amount of traffic will arrive at a website at any given time, and to estimate delays in each stage of the queue, from accepting a request, to waiting to be served, to the time taken to actually be served. We look at what queueing theory entails.

Website Failures

Today we're looking at some server overload circumstances from the sad, to the silly and the very expensive ... and some where the hackers got the blame for something that may well have been the load tester's responsibility!

Website flooded after heavy rains

Load Testing Interview with David Makogon

load testing David MakogonThe following is an interview with David Makogon, and he shares his thoughts with us on load testing.

Load Testing and Climate Change

Anyone know how I can find some cool things that happened in information technology on this day in history? I tried to Google it, but the results are useless. However, I did find out that today is Blog Action Day, which is an annual event held every October 15 that unites the world’s bloggers in posting about the same issue on the same day with the aim of sparking discussion around an issue of global importance.

Load Testing News for October 14

stress testing weddingOn October 14, 1962, six thousand (yes 6,000) Unification church couples got married in Korea. Think about it...that is probably the largest STRESS test of all time! :-)

Before we get to the software testing news, here are some other events that happened on this day in history that I find interesting:

  • Space probe Magellan burns up in atmosphere of Venus in 1994

Software Testing & Paul Simon

software testing paul simonToday is the 67th birthday of a musician who has been relevant in the mainstream for longer than virtually anyone else in modern music. Paul Simon was big when I was a child and still has a strong following. I remember the tunes of Simon and Garfunkel playing in our house before I even knew any of the entertainers. "I am just a poor boy..." was more than just a song, it was my story.

3 Key Load Testing Tips

1. Test against your production environment.

load testing geeksI told this to an "expert" at the Association of Software Testing conference this year. He response was something like, "Are you kidding me?!! You'd have to be stupid to load test your production setup."

What occurred to me later in the conversation was that this guy works for a Fortune 100 company and has a six-figure annual budget (not counting his salary) just for performance engineering.

6 Spectacular Failures - When Performance Testing Goes Very Wrong

Performance testing is important. Web developers never want their sites or applications to crash under load. If they do, so what? Not all of the failures below are related to high traffic, but the failures are notable because of their impact on users and the far-reaching ripple effect.

From promising thousands of customers £7 top-of-the-line computers, to nearly causing an 800-plane pile-up, 'inconvenience' doesn't even begin to describe some of the most famous application failures.

Web Performance Improvement

developer.comI saw a tweet today that referenced web performance improvement, and that linked me to an interesting post on Developer.com.

Load Testing Drupal - Anonymous Users

Drupal logo

A Little Cache Goes a Long Way

Drupal is the target of load testing for this series of articles. If this is your first time reading any of the articles in this series, please review the introduction and summary called Load Testing Drupal for a good context of what we are doing.

Load Testing Drupal

load testing DrupalDrupal is one of the most popular open source content management platforms and is being adopted at ever larger organizations because of its powerful features.

Our team is conducting load testing against copies of Drupal on various sizes of EC2 instances. We intend to establish performance benchmarks for Drupal version 6.x.

Web Application Development - Top 3 Principles

Everybody has problems. As web developers, we are trained to find solutions using technology. I submit to you that the most important things (all great lists come in threes) when solving a web problem are:

  1. Simpler is better
  2. There is always a way
  3. Passion overcomes intelligence and skill


simpler is better

Simpler is Better

Load Testing Feedback Contest

We at LoadStorm are excited to announce that we now have 2,000 customers!

In order to celebrate this milestone, we are hosting a contest for the best LoadStorm feedback. WOULD YOU LIKE TO WIN $250?

We are eager to hear from all of our customers so that we can make LoadStorm a better fit for everyone's load testing needs. Please submit your best LoadStorm recommendations, frustrations, wish list, or any type of feedback that you believe will help our developers improve our product.

New Pricing Model

Storm on Demand Users Cost
250 $9.97
500 $19.95
1,000 $39.90
5,000 $199.50
10,000 $399.00
25,000 $497.50
50,000 $995.00

To See All Plans & Pricing Details

Performance Metrics

  • Average Response Times
  • Peak Response Times
  • Error Rates
  • Throughput
  • Requests per Second
  • Concurrent Users

Web Developers Like Us!

“I deeply resent every second of my life I waste by thinking about load testing. All I ask is that our site be tested with a lot of traffic and without a bunch of BS for me to deal with. Thanks to LoadStorm, I need never again lose another moment to this insufferably tedious aspect of my job. I can feel the rage melting away.” - Shawn Miller, Web Developer, Woot.com