Getting Started Archives | LoadStorm

LoadStorm™ is a web-based load testing tool for simulating what users do with a web site or web application. You use it to build tests that send requests to your server in the same way that a user’s browser sends requests to your server. But these tests are executed by our automated systems rather than by a user, so they can be done repeatedly and in large numbers simultaneously. They can also be built using our tool in such a way as to simulate a large number of different users with different tasks to perform.

It depends on how many simulated users you need for testing, and it depends on if you want Storm on Demand (pay per test) or Monthly Subscription.

The Breeze account is free forever (not just a trial), and it is a fully functional version that can be used for building and running any test plan or scenarios. The only limit of a free account is 25 concurrent users.

Premium subscriptions provide higher levels of load and begin at $39.90. Storm on Demand is approximately $0.0399 per virtual user.

For all the details, please see Load Testing Cost.

Subscription Highlights

• No limit to the number of tests you can run in a month.
• No obligation to subscribe beyond a month.
• Your test plans & results are preserved forever.

Our verification process works exactly like Google Analytics. The idea is to prove it is your server so LoadStorm isn’t used for a denial of service attack. You can put a file with the name generated by LS into your root directory (doesn’t matter what is in it). So let’s say your target server is http://www.xyzcorp.com, then there should be a valid response if a request comes to your server for http://www.xyzcorp.com/loadstorm-90630fc588.html.

Alternatively, if you don’t want to use the file you can embed the code in your default page. To continue the example, the HTML page received from requesting http://www.xyzcorp.com should contain a string somewhere within it (inside a comment tag is good) that matches “loadstorm-90630fc588″. LoadStorm will parse the page returned by your server exactly like a browser will. In that parsing, the verification string must appear somewhere for us to consider it verified.

Once you have the file or text in place, click on the Build Tab, then select the plan name that is targeting your server. You will see a Verify Now link in the yellow shaded section that shows your verification code. Click on that link and you should now see “Verified: Yes” on the plan page. If you have difficulty or need assistance, please contact [email protected].

Please only use the domain name when adding a server, and only use the domain name once. Adding a server for http://www.xyz.com works great and verification is successful. Then creating a server for http://www.xyz.com/myfolder will not work and trying to verify it will produce an error.

Most people create several scenarios to accurately reflect different types of users. Scenarios execute simultaneously in order to hit your site with the correct mix of traffic. You control the weighting of each scenario by giving it a number relative to the others. For example, if you have 2 scenarios and you give one a weighting of 4 and the other a weighting of 1, then the first will generate 80% of the traffic and the second 20%. This 80/20 could be realistic for an retail e-commerce site where most of the people (80%) are browsing the product catalog, while a minority of people (20%) are going through a buying experience. You can of course throw in a low-weight scenario to represent some internal employees adding products or issuing refunds.

There is no limit to the number of scenarios you can create in a test plan. We encourage you to build as many as necessary to accurately reflect the real traffic patterns on your site. That will give you the most meaningful metrics for tuning your applications.

We also simulate the user “think time” with random pauses between steps. Real users click on something, then read the page or fill in a form, and then click on something else. Thus, you don’t want to have a simulated virtual user clicking on something every .5 seconds – it would not give you meaningful results if you want to see how your site holds up in the real world.

Also see: Virtual User Calculations about calculating the amount of virtual users you need for a realistic test.

It helps you with performance engineering. We recommend that you do not begin a large test at the maximum number of virtual users. Rather than run a test for 10 minutes beginning with 5,000 and ending with 5,000, we suggest that you perhaps start with 500 users and increase to 5,000 over an hour. This approach has the advantage of allowing your system to get all the parts working properly at smaller load (e.g. caches, threads, database connections) before the heavier volume starts exposing potential bottlenecks in your application.

From what we’ve seen in thousands of load tests with LoadStorm, it is common to see useful patterns in the metrics before the peak traffic is reached, and these metrics point to areas of performance limitations that can be tuned.

One of the primary reasons we price LoadStorm subscriptions to allow unlimited test runs in the month is because we know that load testing is invariably coupled with performance tuning – an iterative process of test/tune/test/tune. Therefore, we recommend you ramp the volume in order to get more value from the test results.

LoadStorm needs a little time to prepare the test resources such as the load generation servers. Our system is using each of your scenarios and steps to pre-calculate the virtual users’ actions, how many servers will need to be instantiated from the cloud, random pauses, etc. This produces essentially a test roadmap that our Scheduler & Summarizer modules use to coordinate all the http traffic and correctly capture the metrics. Depending on the number of concurrent users and the duration of your test, the preparation of a load test can take 2 to 15 minutes.

Because we use the Amazon EC2 cloud to dynamically instantiate load generation servers, we don’t have a specific static IP address, but here are the ranges:

US East (Northern Virginia):

72.44.32.0/19 (72.44.32.0 – 72.44.63.255)
67.202.0.0/18 (67.202.0.0 – 67.202.63.255)
75.101.128.0/17 (75.101.128.0 – 75.101.255.255)
174.129.0.0/16 (174.129.0.0 – 174.129.255.255)
204.236.192.0/18 (204.236.192.0 – 204.236.255.255)
184.73.0.0/16 (184.73.0.0 – 184.73.255.255)
184.72.128.0/17 (184.72.128.0 – 184.72.255.255)
184.72.64.0/18 (184.72.64.0 – 184.72.127.255)
50.16.0.0/15 (50.16.0.0 – 50.17.255.255)
50.19.0.0/16 (50.19.0.0 – 50.19.255.255)
107.20.0.0/14 (107.20.0.0 – 107.23.255.255)
23.20.0.0/14 (23.20.0.0 – 23.23.255.255)
54.242.0.0/15 (54.242.0.0 – 54.243.255.255)
54.234.0.0/15 (54.234.0.0 – 54.235.255.255)
54.236.0.0/15 (54.236.0.0 – 54.237.255.255)
54.224.0.0/15 (54.224.0.0 – 54.225.255.255)
54.226.0.0/15 (54.226.0.0 – 54.227.255.255)
54.208.0.0/15 (54.208.0.0 – 54.209.255.255)
54.210.0.0/15 (54.210.0.0 – 54.211.255.255)
54.221.0.0/16 (54.221.0.0 – 54.221.255.255)
54.204.0.0/15 (54.204.0.0 – 54.205.255.255)
54.196.0.0/15 (54.196.0.0 – 54.197.255.255)
54.198.0.0/16 (54.198.0.0 – 54.198.255.255)

US West (Northern California):

204.236.128.0/18 (204.236.128.0 – 204.236.191.255)
184.72.0.0/18 (184.72.0.0 – 184.72.63.255)
50.18.0.0/16 (50.18.0.0 – 50.18.255.255)
184.169.128.0/17 (184.160.128.0 – 184.169.255.255)
54.241.0.0/16 (54.241.0.0 – 54.241.255.255)
54.215.0.0/16 (54.215.0.0 – 54.215.255.255)
54.219.0.0/16 (54.219.0.0 – 54.219.255.255)

EU (Ireland):

79.125.0.0/17 (79.125.0.0 – 79.125.127.255)
46.51.128.0/18 (46.51.128.0 – 46.51.191.255)
46.51.192.0/20 (46.51.192.0 – 46.51.207.255)
46.137.0.0/17 (46.137.0.0 – 46.137.127.255)
46.137.128.0/18 (46.137.128.0 – 46.137.191.255)
176.34.128.0/17 (176.34.128.0 – 176.34.255.255)
176.34.64.0/18 (176.34.64.0 – 176.34.127.255)
54.247.0.0/16 (54.247.0.0 – 54.247.255.255)
54.246.0.0/16 (54.246.0.0 – 54.246.255.255)
54.228.0.0/16 (54.228.0.0 – 54.228.255.255)
54.216.0.0/15 (54.216.0.0 – 54.217.255.255)
54.229.0.0/16 (54.229.0.0 – 54.229.255.255)
54.220.0.0/16 (54.220.0.0 – 54.220.255.255)
54.194.0.0/15 (54.194.0.0 – 54.195.255.255)

Asia Pacific (Singapore)

175.41.128.0/18 (175.41.128.0 – 175.41.191.255)
122.248.192.0/18 (122.248.192.0 – 122.248.255.255)
46.137.192.0/18 (46.137.192.0 – 46.137.255.255)
46.51.216.0/21 (46.51.216.0 – 46.51.223.255)
54.251.0.0/16 (54.251.0.0 – 54.251.255.255)
54.254.0.0/16 (54.254.0.0 – 54.254.255.255)
54.255.0.0/16 (54.255.0.0 – 54.255.255.255)

 
We obtained these ranges from Amazon’s EC2 list of public IP ranges.

The Form Data Set field that you see on the Scenario settings page is a mechanism to select an input source for form submission. Using the Form Data Sets functionality, it is possible to submit forms such as a login form and fill-in the fields (e.g. username, password) with data from your system. Other common types of form submission are search terms, credit card numbers, and profile preferences. Any type of form field that can be filled in with text can utilize the Form Data Sets.

You may upload your own data to be used for form submission, or you may use one of our built-in form data sets.

The Tiny, Small, and Huge form data sets are built-into LoadStorm. The only real difference between the three files are the number of rows. Tiny only has 10 records. Small has 100, and Huge has 10,000 records of user information. They contain user information:

• First Name
• Last Name
• Username
• Password
• Email address

Many of our customers use one of our built-in form data sets for registration on their web application or for logging in virtual users.

If you are going to upload your own data into LoadStorm, here is an example of the process for getting user credentials from your system to LoadStorm:

1. Export a CSV file of user information from your user table(s).
2. The first row of the CSV must contain the names of the columns, and these column names will be mapped to the field name in the step that submits the form.
3. The CSV file is uploaded by clicking on the UPLOAD DATA button on the Build tab. This button will be near the bottom, under the Form Data Sets section of the page.
4. You must give the file a label so that you can choose that file to be used at the scenario level.

Once you have the CSV data in LoadStorm (you should see it on the Build tab under the heading Form Data Sets), then you need to edit the scenario and choose the appropriate data set based on the label you assigned when uploading it. This essentially connects the CSV file to your scenario. Then when you add a step for form submission, you will be able to map the fields in the form to the columns of data in the Form Data Set.

The LoadStorm Tour has a video tutorial showing how to create a login form submission. Watch the video on YouTube

When adding a step in LoadStorm, our emulator makes the request to your web application and receives the response containing the HTML code for the page. Our tool then parses the HTML to determine what additional resources (e.g. images, Javascript libraries) need to be requested for that page.

Each of those resources are requested and received. LoadStorm saves all of the HTML and other resources in our database along with other information it needs to reproduce the sequence of events for this step during a load test. The visual rendering you see at the bottom of the step is an iFrame where our tool places a copy of what was received during the step creation process. It is NOT an accurate rendering as you would see it in a full browser without all of the other LoadStorm code and process
“wrapped around” it. Essentially, what you are seeing is a page inside a page. The images may not display because of how they are referenced (e.g. absolute or relative) in the HTML code.

The purpose of this iFrame display is to give the person building test scenarios confidence that the correct requests/responses were transacted for this step. The images you don’t see are still going to be requested during the load test, and all of the performance metrics such as response times will be accurate. Virtual users do not render the pages. No one will actually “see” the transactions during a load test because there is no need to see the pages, and our tool is measuring the Throughput, Error Rate, Response Times, Requests per Second, and Concurrent Users from many different load generation machines that process each virtual user going through the steps.

Occasionally, a test will be terminated early because the target server fails or the tester stops it. It is not uncommon to have a test automatically stopped by LoadStorm when error rates hit 70% or so. When this happens in a test using Storm on Demand virtual users, those users may seem “wasted”.

Customers have asked us why the system deducted 5,000 vusers from their account when the test only lasted 10 minutes and stopped at 500 users. The reason LoadStorm does not prorate the Storm on Demand vusers to the minute or to the actual peak users is because we incur the full costs of the 5,000 vuser test.

Amazon’s cloud is elastic, and we “rent” the servers needed for your tests. However, there is a minimum of 1 hour increments that Amazon will bill us. For example, if your test needs 20 servers to hit your defined peak users, we buy those 20 servers upfront before your test actually starts. Then if you load test dies 5 minutes into it, we are still charged as if the test used all 20 servers for 1 hour.

Thus, we recommend that you start with smaller tests to verify that you have all of the environmental factors properly configured. Growing the volume in several tests that increase in volume will potentially eliminate the “wasted” feeling of throwing 5,000 users at a server that fails at 500 users.

For some projects, this is an acceptable approach. For some internal applications, apps that are very simple or realistically will expect light traffic, the best plan may be to get it done as quickly as possible. You have to ask yourself what is the risk of the application not working at some point. For many businesses, getting a large influx of new users is the peak of success, and that success would be shattered by having the new users unable to use the system, unable to buy your product and unhappy with your company.

Yes, you may add Storm on Demand users for any load test. This allows you to spike test traffic to a much higher level than your subscription.

For example, if you have a Hurricane subscription for 5,000 maximum concurrent vusers, you can purchase an additional 10,000 Storm on Demand users for a total of 15,000 vusers on a single test. Those 10,000 on demand users would be available for only one load test, and the 5,000 are available for other tests throughout the month.

Yes, and only if you want to. LoadStorm has a checkbox on the scenario edit page to turn download images on or off. There is a similar option for controlling the download and execution of Javascript files.

Normally performance issues arise in the process of generating the dynamic page, and web servers can serve static images very quickly; however, we understand that many of our clients have websites with a large amount of media-related content, and load test results from those sites could be significantly affected by not downloading that content. Image download is especially relevant to performance if bottlenecks are in bandwidth or other network communications.