Occasionally, a test will be terminated early because the target server fails. It is not uncommon to have a test automatically stopped by LoadStorm when error rates hit 70% or so. When this happens in a test using Storm on Demand virtual users, those users may seem “wasted”.
Customers have asked us why the system deducted 5,000 vusers from their account when the test only lasted 10 minutes and stopped at 500 users. The reason LoadStorm does not prorate the Storm on Demand vusers to the minute or to the actual peak users is because we incur the full costs of the 5,000 vuser test.
Amazon’s cloud is elastic, and we “rent” the servers needed for your tests. However, there is a minimum of 1 hour increments that Amazon will bill us. For example, if your test needs 20 servers to hit your defined peak users, we buy those 20 servers upfront before your test actually starts. Then if you load test dies 5 minutes into it, we are still charged as if the test used all 20 servers for 1 hour.
Thus, we recommend that you start with smaller tests to verify that you have all of the environmental factors properly configured. Growing the volume in several tests that increase in volume will potentially eliminate the “wasted” feeling of throwing 5,000 users at a server that fails at 500 users.