Please feel free to contact us via phone or email for more detailed information about LoadStorm.com.

What does it do?

LoadStorm™ is a web-based load testing tool for simulating what users do with a web site or web application. You use it to build tests that send requests to your server in the same way that a user's browser sends requests to your server. But these tests are executed by our automated systems rather than by a user, so they can be done repeatedly and in large numbers simultaneously. They can also be built using our tool in such a way as to simulate a large number of different users with different tasks to perform.

Why do I need to verify my server?

Our verification process works exactly like Google Analytics. The idea is to prove it is your server so LoadStorm isn't used for a denial of service attack. You can put a file with the name generated by LS into your root directory (doesn't matter what is in it). So let's say your target server is http://www.xyzcorp.com, then there should be a valid response if a request comes to your server for http://www.xyzcorp.com/loadstorm-90630fc588.html.

Alternatively, if you don't want to use the file you can embed the code in your default page. To continue the example, the HTML page received from requesting http://www.xyzcorp.com should contain a string somewhere within it (inside a comment tag is good) that matches "loadstorm-90630fc588". LoadStorm will parse the page returned by your server exactly like a browser will. In that parsing, the verification string must appear somewhere for us to consider it verified.

Once you have the file or text in place, click on the Build Tab, then select the plan name that is targeting your server. You will see a Verify Now link in the yellow shaded section that shows your verification code. Click on that link and you should now see "Verified: Yes" on the plan page. If you have difficulty or need assistance, please contact support@loadstorm.com.

When configuring a scenario step that clicks a link, how does LoadStorm follow the link?

LoadStorm follows links when the session ID is appended to the URL query string because it searches for the text between the tags. If it doesn't find that text, then the fall back is to match the XPATH. We stored the XPATH when the scenario was created (essentially an address inside the page's DOM). The XPATH is a unique identifier that allows us to click on the anchor such as an image or the href attribute or execute the Javascript.

What does "response time" mean?

In LoadStorm, the response time would be the amount of time taken for the "primary page" to fully download. The primary page normally includes the HTML, CSS, and Javascript files. However, the primary page could be another resource such a PDF, video, or Flash file. The response time is likely to be faster than observed by you in a browser because it does not include downloading images or time for a browser to load and render the page.

It is also important to note that LoadStorm's servers probably have faster internet connections than most homes and offices.

Does LoadStorm download images as a part of the load testing?

Only if you want to. LoadStorm has a checkbox on the scenario edit page to turn download images on or off. There is a similar option for controlling the download and execution of Javascript files.

Normally performance issues arise in the process of generating the dynamic page, and web servers can serve static images very quickly; however, we understand that many of our clients have websites with a large amount of media-related content, and load test results from those sites could be significantly affected by not downloading that content. Image download is especially relevant to performance if bottlenecks are in bandwidth or other network communications.

While I am running a load test and X number of users are hitting my web app, does this mean every second X number of requests are being sent to my application?

The short answer is that a virtual or simulated user does what a "real" user does as specified by the scenarios and steps that you have created in LoadStorm. If there are X number of users, then there are X scenarios running at that particular time.

When creating a scenario, you can adjust the pause times which is how much time there is between steps. The actual pause is randomly generated to give a more realistic load against the server. As an example, a scenario which opens a single page for each step with an average pause of 30 seconds will have an average of 2 requests per minute per user. It gets a bit more complicated when there are external files like Javascript. These are added as well and counted in the total requests. When one scenario ends for a designated number of users, another is started within a minute to maintain the desired user load until the end of the test.

What's the difference between "Concurrent Users", "Sessions", and "Total Users"?

The starting and peak number of virtual users in LoadStorm represent concurrent users or the number of simulated users at a particular point in time. Each concurrent user will last for the duration of the scenario. If you have only one step in your scenario, then a concurrent user will last for less than a minute and then another one will take its place.

For example, if you have a test with one scenario having one step, starting at 10 concurrent users and peak of 10 concurrent users for 20 minutes, there will be 200-300 total users simulated (and same number of sessions). LoadStorm will create a new session for each virtual user, but it will not maintain the session after the scenario has ended, so your session should time out as you have configured it.

The LoadStorm test parameters simulate "concurrent" users not total users. The total simulated users will generally be much higher than concurrent users, but the actual count depends on lots of other factors such as the number of steps in the scenarios, pause times, etc.

How should I setup scenarios to simulate real traffic?

Most people create several scenarios to accurately reflect different types of users. Scenarios execute simultaneously in order to hit your site with the correct mix of traffic. You control the weighting of each scenario by giving it a number relative to the others. For example, if you have 2 scenarios and you give one a weighting of 4 and the other a weighting of 1, then the first will generate 80% of the traffic and the second 20%. This 80/20 could be realistic for an retail e-commerce site where most of the people (80%) are browsing the product catalog, while a minority of people (20%) are going through a buying experience. You can of course throw in a low-weight scenario to represent some internal employees adding products or issuing refunds.

There is no limit to the number of scenarios you can create in a test plan. We encourage you to build as many as necessary to accurately reflect the real traffic patterns on your site. That will give you the most meaningful metrics for tuning your applications.

We also simulate the user "think time" with random pauses between steps. Real users click on something, then read the page or fill in a form, and then click on something else. Thus, you don't want to have a simulated virtual user clicking on something every .5 seconds - it would not give you meaningful results if you want to see how your site holds up in the real world.

Does loadstorm use a browser for accessing the websites to test or does it emulate a browser?

LoadStorm emulates a browser. If we used an actual browser, we would trade-off the cost effectiveness of our solution for the ability to actually process exactly as a browser. Most solutions use actual browsers and cost you about 200-1,000% more per user/hour of testing. We understand some people need the real browsers and are willing to pay for it. We are trying to help those that achieve their load testing goals with emulation and a much lower cost. We do handle most caching and Ajax processing, but sometimes a customer will report a problem; whereupon our developers try to immediately change our system to accommodate the customer's need.

How does it generate HTTP traffic against my web application?

LoadStorm is a truly distributed application that leverages the power of Amazon Web Services to scale on demand with processing power and bandwidth as needed to test the largest web projects. As you crank up the testing load to 200 hundred or 20,000 virtual users, LoadStorm is seamlessly adding machines from Amazon's server farm to handle the processing. When your tests are done and the extra machines are not needed, they are turned off to wait for another day. Thanks to Amazon for putting together this great package of services including S3, EC2, SQS and others.

What are the IP addresses ranges used by LoadStorm? We need to white list them.

Because we use Amazon EC2 cloud to dynamically instantiate load generation servers, we don't have a specific static IP address, but here are the ranges:

216.182.224.0/20 (216.182.224.0 - 216.182.239.255)
72.44.32.0/19 (72.44.32.0 - 72.44.63.255)
67.202.0.0/18 (67.202.0.0 - 67.202.63.255)
75.101.128.0/17 (75.101.128.0 - 75.101.255.255)
174.129.0.0/16 (174.129.0.0 - 174.129.255.255)

Other than IP address, is there any way to identify LoadStorm requests from normal traffic? User agent? Other HTTP header?

Yes, the user agent includes LoadStorm with the version number (currently 1.5).

Can your system run load tests behind my firewall for non-Internet applications?

Unfortunately not. LoadStorm is only for web applications that can be reached from the Amazon cloud.

Could you please clarify whether or not the load testing should show up in the Google Analytics reports, and how the testing is typically tracked by Google Analytics?

We would not expect LoadStorm test volume to show up in your Google Analytics. We do execute the javascript, but requests outside of your domain are blocked unless those other servers have also been verified. Servers like Google Analytics should be set to "Ignore". So, the tracking by Analytics, ad servers or other systems outside of your domain would not work. Some of these kinds of services have restrictions against load testing.

Why should I ramp up my test load volume over time?

It helps you with performance engineering. We recommend that you do not begin a large test at the maximum number of virtual users. Rather than run a test for 10 minutes beginning with 5,000 and ending with 5,000, we suggest that you perhaps start with 500 users and increase to 5,000 over an hour. This approach has the advantage of allowing your system to get all the parts working properly at smaller load (e.g. caches, threads, database connections) before the heavier volume starts exposing potential bottlenecks in your application.

From what we've seen in thousands of load tests with LoadStorm, it is common to see useful patterns in the metrics before the peak traffic is reached, and these metrics point to areas of performance limitations that can be tuned.

One of the primary reasons we price LoadStorm subscriptions to allow unlimited test runs in the month is because we know that load testing is invariably coupled with performance tuning - an iterative process of test/tune/test/tune. Therefore, we recommend you ramp the volume in order to get more value from the test results.

How do I stop a test that has killed my server?"

You can press the "STOP NOW" button at any time. It is found on the Run tab, and the button will only appear during the execution of a load test.

Also, LoadStorm will automatically stop a load test if it can determine that your server has been overloaded. The thresholds for stopping a test are currently when the Error Rate exceeds 50% or the Average Response Time exceeds 15 seconds for the most recently summarized test period.

Important Note for Small Tests: The sample interval is not the entire test but just the most recent part of it, and there must be a minimum number of requests in the sample. This interval is significant because in very small tests the number of failed requests AND the total requests may be few. For example, you must have some requests for at least 2 intervals, which means you have summarized more than 1 minute of requests.

Why doesn't my test start immediately?

LoadStorm is using each of your scenarios and steps to pre-calculate the virtual users' actions, how many servers will need to be instantiated from the cloud, random pauses, etc. This produces essentially a test roadmap that our Scheduler & Summarizer modules use to coordinate all the http traffic and correctly capture the metrics. Depending on the number of concurrent users and the duration of your test, the pre-calculation process of preparing a load test can take 2 to 15 minutes.

How much does it cost?

It depends on how many simulated users you need for testing, and it depends on if you want Storm on Demand (pay per test) or Monthly Subscription.

The Breeze account is free forever (not just a trial), and it is a fully functional version that can be used for building and running any test plan or scenarios. The only limit of a free account is 25 concurrent users.

Premium subscriptions provide higher levels of load and begin at $19.99. Storm on Demand is approximately $0.039 per virtual user.

For all the details, please see Load Testing Cost.

Subscription Highlights

• No limit to the number of tests you can run in a month.
• No obligation to subscribe beyond a month.
• Your test plans & results are preserved forever.

Why not let our users test the system's capacity?

For some projects, this is an acceptable approach. For some internal applications, apps that are very simple or realistically will expect light traffic, the best plan may be to get it done as quickly as possible. You have to ask yourself what is the risk of the application not working at some point. For many businesses, getting a large influx of new users is the peak of success, and that success would be shattered by having the new users unable to use the system, unable to buy your product and unhappy with your company.