“Wasted” Users When the Target Fails – LoadStorm

We beg our customers to consider iterative load testing cycles that begin with small volumes your web application can handle easily. Then, gradually increase the peak volume in subsequent test cycles until you find the breaking point.

Occasionally a test will be terminated early because the target server fails.  There is a STOP TEST button on the page for a running test, and sometimes a customer will press the button because the Error Rate and Response Times indicate a failure condition – there is no need to continue the test.

The test didn’t reach the Peak VUsers scheduled. When this happens in a test using Storm on Demand virtual users, those users may seem “wasted”.

We All Lose If Your Test Fails Early

Customers have asked us why the system deducted 5,000 VUsers from their account when the test only lasted 10 minutes and they stopped it at 500 users. They feel the other 4,500 VUsers were “unused” and should be refunded. However, from LoadStorm’s point of view, the load testing tool did everything it could to deliver the VUsers because it instantiated all the servers and prepared the full 5,000 VUser test.

The reason LoadStorm does not prorate the Storm on Demand VUsers to the minute or to the actual peak users is because we incur the full costs of the 5,000 VUser test.

Amazon’s cloud is elastic, and we “rent” the servers needed for your tests. However, there is a minimum of 1 hour increments that Amazon will bill us. For example, if your test needs 20 servers to hit your defined peak users, we buy those 20 servers upfront before your test actually starts. Then if your load test dies 5 minutes into it, we are still charged as if the test used all 20 servers for 1 hour.

“My Website Can Easily Stand-up to 50,000 Users”

That is a direct quote from an overly confident customer.  Their project didn’t work out as planned.

We have heard this several times. We warned them to start with a test peaking at 5,000 or 10,000 VUsers before trying to stress the system with the full load. Nope. They were confident. They didn’t want to run more than one test. They were stubborn, and they were wrong.

One customer’s website failed within the first 15 seconds because he started at 50,000 VUsers. We felt sorry for him, but we did beg him not to run the test with such a high starting volume. He learned very little from the test. He didn’t know if his site would perform well at 1,000 VUsers and at what load it would start slowing down or show errors. All he knew for sure was that it died immediately under load of 50,000 concurrent users. Please learn from his mistake!

Please Ramp Your Tests

Thus, we recommend that you start with smaller tests to verify that you have all of the environmental factors properly configured. Growing the volume in several tests that increase in volume will potentially eliminate the “wasted” feeling of throwing 5,000 users at a server that fails at 1,000 users.

Also, use the linear pattern to ramp your volume steadily throughout a 60-minute test because if your site fails, you will have a much better measurement of what load causes response times to increase or errors to be returned by your web server. Please heed this advice. Call our support team if you want to discuss the merits of this best practice.