FAQ » LoadStorm LoadStorm

LoadStorm™ is a web-based load testing tool for simulating what users do with a web site or web application. You use it to build tests that send requests to your server in the same way that a user’s browser sends requests to your server. But these tests are executed by our automated systems rather than by a user, so they can be done repeatedly and in large numbers simultaneously. They can also be built using our tool in such a way as to simulate a large number of different users with different tasks to perform.

It depends on how many simulated users you need for testing, and it depends on if you want Storm on Demand (pay per test) or Monthly Subscription.

The Breeze account is free forever (not just a trial), and it is a fully functional version that can be used for building and running any test plan or scenarios. The only limit of a free account is 25 concurrent users.

Premium subscriptions provide higher levels of load and begin at $39.90. Storm on Demand is approximately $0.0399 per virtual user.

For all the details, please see Load Testing Cost.

Subscription Highlights

• No limit to the number of tests you can run in a month.
• No obligation to subscribe beyond a month.
• Your test plans & results are preserved forever.

Our verification process works exactly like Google Analytics. The idea is to prove it is your server so LoadStorm isn’t used for a denial of service attack. You can put a file with the name generated by LS into your root directory (doesn’t matter what is in it). So let’s say your target server is http://www.xyzcorp.com, then there should be a valid response if a request comes to your server for http://www.xyzcorp.com/loadstorm-90630fc588.html.

Alternatively, if you don’t want to use the file you can embed the code in your default page. To continue the example, the HTML page received from requesting http://www.xyzcorp.com should contain a string somewhere within it (inside a comment tag is good) that matches “loadstorm-90630fc588″. LoadStorm will parse the page returned by your server exactly like a browser will. In that parsing, the verification string must appear somewhere for us to consider it verified.

Once you have the file or text in place, click on the Build Tab, then select the plan name that is targeting your server. You will see a Verify Now link in the yellow shaded section that shows your verification code. Click on that link and you should now see “Verified: Yes” on the plan page. If you have difficulty or need assistance, please contact [email protected].

Please only use the domain name when adding a server, and only use the domain name once. Adding a server for http://www.xyz.com works great and verification is successful. Then creating a server for http://www.xyz.com/myfolder will not work and trying to verify it will produce an error.

Most people create several scenarios to accurately reflect different types of users. Scenarios execute simultaneously in order to hit your site with the correct mix of traffic. You control the weighting of each scenario by giving it a number relative to the others. For example, if you have 2 scenarios and you give one a weighting of 4 and the other a weighting of 1, then the first will generate 80% of the traffic and the second 20%. This 80/20 could be realistic for an retail e-commerce site where most of the people (80%) are browsing the product catalog, while a minority of people (20%) are going through a buying experience. You can of course throw in a low-weight scenario to represent some internal employees adding products or issuing refunds.

There is no limit to the number of scenarios you can create in a test plan. We encourage you to build as many as necessary to accurately reflect the real traffic patterns on your site. That will give you the most meaningful metrics for tuning your applications.

We also simulate the user “think time” with random pauses between steps. Real users click on something, then read the page or fill in a form, and then click on something else. Thus, you don’t want to have a simulated virtual user clicking on something every .5 seconds – it would not give you meaningful results if you want to see how your site holds up in the real world.

Also see: Virtual User Calculations about calculating the amount of virtual users you need for a realistic test.

It helps you with performance engineering. We recommend that you do not begin a large test at the maximum number of virtual users. Rather than run a test for 10 minutes beginning with 5,000 and ending with 5,000, we suggest that you perhaps start with 500 users and increase to 5,000 over an hour. This approach has the advantage of allowing your system to get all the parts working properly at smaller load (e.g. caches, threads, database connections) before the heavier volume starts exposing potential bottlenecks in your application.

From what we’ve seen in thousands of load tests with LoadStorm, it is common to see useful patterns in the metrics before the peak traffic is reached, and these metrics point to areas of performance limitations that can be tuned.

One of the primary reasons we price LoadStorm subscriptions to allow unlimited test runs in the month is because we know that load testing is invariably coupled with performance tuning – an iterative process of test/tune/test/tune. Therefore, we recommend you ramp the volume in order to get more value from the test results.

LoadStorm needs a little time to prepare the test resources such as the load generation servers. Our system is using each of your scenarios and steps to pre-calculate the virtual users’ actions, how many servers will need to be instantiated from the cloud, random pauses, etc. This produces essentially a test roadmap that our Scheduler & Summarizer modules use to coordinate all the http traffic and correctly capture the metrics. Depending on the number of concurrent users and the duration of your test, the preparation of a load test can take 2 to 15 minutes.

LoadStorm emulates a browser. If we used an actual browser, we would trade-off the cost effectiveness of our solution for the ability to actually process exactly as a browser. Some solutions use actual browsers and cost you about 200-1,000% more per Virtual User Hour of testing.

We understand some web developers need the real browsers and are willing to pay for it. Our load testing tool is designed to help those developers that achieve their load testing goals with emulation and a much lower cost.

LoadStorm’s emulator makes the requests just like a browser, and it parses the HTML received in the response. As it parses the page, it identifies the additional resources needed such as images, stylesheets, XML, and Javascript library files. LoadStorm will automatically send requests to your server for these files. Each response is measured and recorded for speed and status codes.

Our emulator will respect caching. For example, if you have the same logo image on every page, LoadStorm will only request it once for each virtual user, and subsequent steps will request any new images but not the ones already returned by your server.

LoadStorm will process the Javascript that is triggered on page load. However, our emulator will not process background requests of Javascript or Ajax. We acknowledge the inability to process Ajax is a limitation of our tool, and we recognize that many Rich Internet Applications cannot use our tool for testing. We are working on a new solution that will remove this limitation.

Because we use the Amazon EC2 cloud to dynamically instantiate load generation servers, we don’t have a specific static IP address, but here are the ranges:

US East (Northern Virginia):

72.44.32.0/19 (72.44.32.0 – 72.44.63.255)
67.202.0.0/18 (67.202.0.0 – 67.202.63.255)
75.101.128.0/17 (75.101.128.0 – 75.101.255.255)
174.129.0.0/16 (174.129.0.0 – 174.129.255.255)
204.236.192.0/18 (204.236.192.0 – 204.236.255.255)
184.73.0.0/16 (184.73.0.0 – 184.73.255.255)
184.72.128.0/17 (184.72.128.0 – 184.72.255.255)
184.72.64.0/18 (184.72.64.0 – 184.72.127.255)
50.16.0.0/15 (50.16.0.0 – 50.17.255.255)
50.19.0.0/16 (50.19.0.0 – 50.19.255.255)
107.20.0.0/14 (107.20.0.0 – 107.23.255.255)
23.20.0.0/14 (23.20.0.0 – 23.23.255.255)
54.242.0.0/15 (54.242.0.0 – 54.243.255.255)
54.234.0.0/15 (54.234.0.0 – 54.235.255.255)
54.236.0.0/15 (54.236.0.0 – 54.237.255.255)
54.224.0.0/15 (54.224.0.0 – 54.225.255.255)
54.226.0.0/15 (54.226.0.0 – 54.227.255.255)
54.208.0.0/15 (54.208.0.0 – 54.209.255.255)
54.210.0.0/15 (54.210.0.0 – 54.211.255.255)
54.221.0.0/16 (54.221.0.0 – 54.221.255.255)

US West (Northern California):

204.236.128.0/18 (204.236.128.0 – 204.236.191.255)
184.72.0.0/18 (184.72.0.0 – 184.72.63.255)
50.18.0.0/16 (50.18.0.0 – 50.18.255.255)
184.169.128.0/17 (184.160.128.0 – 184.169.255.255)
54.241.0.0/16 (54.241.0.0 – 54.241.255.255)
54.215.0.0/16 (54.215.0.0 – 54.215.255.255)
54.219.0.0/16 (54.219.0.0 – 54.219.255.255)

EU (Ireland):

79.125.0.0/17 (79.125.0.0 – 79.125.127.255)
46.51.128.0/18 (46.51.128.0 – 46.51.191.255)
46.51.192.0/20 (46.51.192.0 – 46.51.207.255)
46.137.0.0/17 (46.137.0.0 – 46.137.127.255)
46.137.128.0/18 (46.137.128.0 – 46.137.191.255)
176.34.128.0/17 (176.34.128.0 – 176.34.255.255)
176.34.64.0/18 (176.34.64.0 – 176.34.127.255)
54.247.0.0/16 (54.247.0.0 – 54.247.255.255)
54.246.0.0/16 (54.246.0.0 – 54.246.255.255)
54.228.0.0/16 (54.228.0.0 – 54.228.255.255)
54.216.0.0/15 (54.216.0.0 – 54.217.255.255)
54.229.0.0/16 (54.229.0.0 – 54.229.255.255)
54.220.0.0/16 (54.220.0.0 – 54.220.255.255)

Asia Pacific (Singapore)

175.41.128.0/18 (175.41.128.0 – 175.41.191.255)
122.248.192.0/18 (122.248.192.0 – 122.248.255.255)
46.137.192.0/18 (46.137.192.0 – 46.137.255.255)
46.51.216.0/21 (46.51.216.0 – 46.51.223.255)
54.251.0.0/16 (54.251.0.0 – 54.251.255.255)
54.254.0.0/16 (54.254.0.0 – 54.254.255.255)
54.255.0.0/16 (54.255.0.0 – 54.255.255.255)

Yes, there are two additional ways for you to white list our traffic:

1. The user agent in the http header includes LoadStorm. However, if your test scenarios contain steps with https requests, then those headers will be encrypted.
2. If you are not testing against your production server, we recommend trying a non-standard port number to identify the traffic as load testing. LoadStorm allows you to use any port number and can be designated for each target server.

LoadStorm follows links when the session ID is appended to the URL query string because it searches for the text between the open and close anchor tags. If it doesn’t find that text, then the fall back is to match the XPATH. We stored the XPATH when the scenario was created (essentially an address inside the page’s DOM). The XPATH is a unique identifier that allows us to click on the anchor such as an image or the href attribute or execute the Javascript.

Average Duration is the time taken for ALL steps to complete (see step duration calculation), plus the average pause times between steps. Thus, it is the mean time for a specific vuser’s “life”. The actual duration fluctuates with the amount of pause assigned to that vuser. The Average Duration is shown above the steps on the Scenario page.

Duration is the time taken for a step to complete, which is the sum of time to last byte for each request in the step (including HTML, JS, CSS, images, etc.). The Average Duration shown above the steps on the Scenario page is the total of all step durations plus the average pause times between steps.

In LoadStorm the response time is calculated thus: The total time taken between the first byte of the request sent and the last byte of the response received for all requests made. This includes the network latency and the time for the server to respond.

The primary page normally includes the HTML, CSS, and Javascript files. However, the primary page could be another resource such as a PDF, video, or Flash file. The response time may appear slightly faster than observed by you in a browser because it does not include time for a browser to render the page. Of course render time will be affected by browser type, operating system, and horsepower of the client computer.

It is also important to note that LoadStorm’s servers probably have faster internet connections than many homes and offices.

Yes, you may submit forms such as a login form and fill-in the fields (e.g. username, password) with data from your system.

First, you export a CSV file of user information from your user table(s). The first row of the CSV must contain the names of the columns, and these column names will be mapped to the field name in the step that submits the form.

The CSV file is uploaded by clicking on the UPLOAD DATA button on the Build tab. This button will be near the bottom, under the Form Data Sets section of the page. You must give the file a label so that you can choose that file to be used at the scenario level.

If you need to embed a comma inside a particular data field, simply place quotation marks around all field data in the file.

The LoadStorm Tour has a video tutorial showing how to create a login form submission.

LoadStorm is a truly distributed application that leverages the power of Amazon Web Services to scale on demand with processing power and bandwidth as needed to test the largest web projects. As you crank up the testing load to 200 hundred or 20,000 virtual users, LoadStorm is automatically adding machines for you (as many as necessary) from Amazon’s server farm to handle the processing. When your tests are done and the extra machines are not needed, they are turned off to wait for another test.

The normal processing is for a scenario to be processed as a sequential series of steps that you have defined in the test plan. Once a step finishes, the vuser pauses for the specified time (between min and max for scenario), then the next step proceeds. After the last step completes, the vuser context (session, cache, cookies) is cleared and the scenario starts from the beginning again with a new vuser.

You define the number of concurrent users you want, start and peak, during your test. As one vuser completes the scenario as described above, then another vuser is created to replace it – starting at step 1 again.

If you have started at a small number and set the peak number much higher, then LoadStorm will not only retire vusers that have completed the scenario, it will also add additional users throughout the test to ramp up the volume.
This is repeated until the test is done as defined by the test duration field (in minutes) when you began the load test.

The Form Data Set field that you see on the Scenario settings page is a mechanism to select an input source for form submission. Using the Form Data Sets functionality, it is possible to submit forms such as a login form and fill-in the fields (e.g. username, password) with data from your system. Other common types of form submission are search terms, credit card numbers, and profile preferences. Any type of form field that can be filled in with text can utilize the Form Data Sets.

You may upload your own data to be used for form submission, or you may use one of our built-in form data sets.

The Tiny, Small, and Huge form data sets are built-into LoadStorm. The only real difference between the three files are the number of rows. Tiny only has 10 records. Small has 100, and Huge has 10,000 records of user information. They contain user information:

• First Name
• Last Name
• Username
• Password
• Email address

Many of our customers use one of our built-in form data sets for registration on their web application or for logging in virtual users.

If you are going to upload your own data into LoadStorm, here is an example of the process for getting user credentials from your system to LoadStorm:

1. Export a CSV file of user information from your user table(s).
2. The first row of the CSV must contain the names of the columns, and these column names will be mapped to the field name in the step that submits the form.
3. The CSV file is uploaded by clicking on the UPLOAD DATA button on the Build tab. This button will be near the bottom, under the Form Data Sets section of the page.
4. You must give the file a label so that you can choose that file to be used at the scenario level.

Once you have the CSV data in LoadStorm (you should see it on the Build tab under the heading Form Data Sets), then you need to edit the scenario and choose the appropriate data set based on the label you assigned when uploading it. This essentially connects the CSV file to your scenario. Then when you add a step for form submission, you will be able to map the fields in the form to the columns of data in the Form Data Set.

The LoadStorm Tour has a video tutorial showing how to create a login form submission. Watch the video on YouTube

Here are some thoughts on increasing RPS in LoadStorm tests:

1. Reducing pause time between steps (at scenario level) – minimum is 10 second pause, so set both min & max to 10.
2. Increase the number of concurrent users.
3. Reduce the response time from your server which decreases LoadStorm’s waiting for a response.

RPS will always be impacted by the types of pages/resources your scenarios are requesting. The way it works is that LoadStorm cranks up the starting concurrent users shortly after test starts. It then walks each user through the scenario assigned to it. One step is executed (such as home page), and all the images, JS, CSS, XML are requested separately. When those responses are all returned, then the system pauses a random time between min & max pause, then it requests the next step. That continues until the end of the scenario, whereupon that virtual user is retired. After a few seconds, another vuser is created and the process starts again. So, having more users going at the same time, without waiting more than a few hundred ms to get a response, will increase RPS.

We have a mechanism to “import” URLs, and we have a video tutorial that may be helpful – http://youtu.be/5APaPn2zo6M

The concept is to upload a file of strings that can be substituted in a single step of a scenario. Most customers use this functionality to append a query string on the end of a domain. You can produce a CSV with one or more columns containing the strings. Then you add a step that opens a page, and there is specific syntax for mapping the string from a column of the CSV into the URL requested by the step.

It may be helpful to review the Here’s an example of what would be used for the step:

http://cia-factbook.s3.amazonaws.com#{FileOfPaths.pathFull1}

The hash and open brace are significant because they tell LoadStorm to substitute the text. “FileOfPaths” is the label for the CSV I uploaded into my LoadStorm account. “pathFull1″ is the column name inside FileOfPaths that contains the string I want to append on the domain.

To upload the CSV (form data set) of your URLs/query strings into LoadStorm:

1. Export a CSV file with one or more columns containing the URLs.
2. The first row of the CSV must contain the names of the columns, and these column names will be mapped to the substitution point in the step indicated by the #{ } delimiters.
3. The CSV file is uploaded by clicking on the UPLOAD DATA button on the Build tab. This button will be near the bottom, under the Form Data Sets section of the page.
4. You must give the file a label so that you can choose that file to be used at the scenario level.

Once you have the CSV data in LoadStorm (you should see it on the Build tab under the heading Form Data Sets), then you need to edit the scenario and choose the appropriate data set based on the label you assigned when uploading it. This essentially connects the CSV file to your scenario. Then when you add a step for using the URL/query string, you will be able to place #{formDataSet.columnName} from the Form Data Set.

When adding a step in LoadStorm, our emulator makes the request to your web application and receives the response containing the HTML code for the page. Our tool then parses the HTML to determine what additional resources (e.g. images, Javascript libraries) need to be requested for that page.

Each of those resources are requested and received. LoadStorm saves all of the HTML and other resources in our database along with other information it needs to reproduce the sequence of events for this step during a load test. The visual rendering you see at the bottom of the step is an iFrame where our tool places a copy of what was received during the step creation process. It is NOT an accurate rendering as you would see it in a full browser without all of the other LoadStorm code and process
“wrapped around” it. Essentially, what you are seeing is a page inside a page. The images may not display because of how they are referenced (e.g. absolute or relative) in the HTML code.

The purpose of this iFrame display is to give the person building test scenarios confidence that the correct requests/responses were transacted for this step. The images you don’t see are still going to be requested during the load test, and all of the performance metrics such as response times will be accurate. Virtual users do not render the pages. No one will actually “see” the transactions during a load test because there is no need to see the pages, and our tool is measuring the Throughput, Error Rate, Response Times, Requests per Second, and Concurrent Users from many different load generation machines that process each virtual user going through the steps.

Unfortunately, our load testing tool doesn’t have a feature for what you need. Normally this is a simple process of putting a domain name to IP address mapping in a configuration file on a particular machine. The process is not as easy in our situation because we use dynamically instantiated servers from the Amazon cloud; thus, we do not know what IP addresses we will be using before you run your tests. We would need to create new images specifically for your tests, and we would need to make changes to our core launching software that instantiates the images so that new machines are launched with this particular host mapping. If you would like us to coordinate this custom process, please contact our support. Additional charges per test will apply.

Occasionally, a test will be terminated early because the target server fails or the tester stops it. It is not uncommon to have a test automatically stopped by LoadStorm when error rates hit 70% or so. When this happens in a test using Storm on Demand virtual users, those users may seem “wasted”.

Customers have asked us why the system deducted 5,000 vusers from their account when the test only lasted 10 minutes and stopped at 500 users. The reason LoadStorm does not prorate the Storm on Demand vusers to the minute or to the actual peak users is because we incur the full costs of the 5,000 vuser test.

Amazon’s cloud is elastic, and we “rent” the servers needed for your tests. However, there is a minimum of 1 hour increments that Amazon will bill us. For example, if your test needs 20 servers to hit your defined peak users, we buy those 20 servers upfront before your test actually starts. Then if you load test dies 5 minutes into it, we are still charged as if the test used all 20 servers for 1 hour.

Thus, we recommend that you start with smaller tests to verify that you have all of the environmental factors properly configured. Growing the volume in several tests that increase in volume will potentially eliminate the “wasted” feeling of throwing 5,000 users at a server that fails at 500 users.

For some projects, this is an acceptable approach. For some internal applications, apps that are very simple or realistically will expect light traffic, the best plan may be to get it done as quickly as possible. You have to ask yourself what is the risk of the application not working at some point. For many businesses, getting a large influx of new users is the peak of success, and that success would be shattered by having the new users unable to use the system, unable to buy your product and unhappy with your company.

You can press the “STOP NOW” button at any time. It is found on the Run tab, and the button will only appear during the execution of a load test.

Also, LoadStorm will automatically stop a load test if it can determine that your server has been overloaded. The thresholds for stopping a test are currently when the Error Rate exceeds 50% or the Average Response Time exceeds 15 seconds for the most recently summarized test period.

Important Note for Small Tests: The sample interval is not the entire test but just the most recent part of it, and there must be a minimum number of requests in the sample. This interval is significant because in very small tests the number of failed requests AND the total requests may be few. For example, you must have some requests for at least 2 intervals, which means you have summarized more than 1 minute of requests.

If one of your web pages contains a reference to an image that is not available on your system, then a response header will be sent to LoadStorm have Status Code = 404. Additionally, your web server will send an HTML page indicating the 404 “Not Found” page to alert the user. That’s standard practice in web applications.

The reason LoadStorm reports the 404 errors as HTML is because your web server is sending our system the Content Type = “HTML” attribute. LoadStorm separates the data in the HTML and Other rows by the Content Type.

Yes, you may add Storm on Demand users for any load test. This allows you to spike test traffic to a much higher level than your subscription.

For example, if you have a Hurricane subscription for 5,000 maximum concurrent vusers, you can purchase an additional 10,000 Storm on Demand users for a total of 15,000 vusers on a single test. Those 10,000 on demand users would be available for only one load test, and the 5,000 are available for other tests throughout the month.

A round robin load balancer will not work very well for testing your configuration unless you are using a much higher number of virtual users (at least 2000 or so). Each of our simulation servers will resolve the DNS name independently of the others, but each of our servers will simulate hundreds of users. I would expect a peak load of 500 users to get routed to more than one of your servers, but probably not all 5 of them. A peak load of 100 users or less is likely to hit only one of your servers.

We would not expect LoadStorm test volume to show up in your Google Analytics. We do execute the javascript, but requests outside of your domain are blocked unless those other servers have also been verified. Servers like Google Analytics should be set to “Ignore”. So, the tracking by Analytics, ad servers or other systems outside of your domain would not work. Some of these kinds of services have restrictions against load testing.

Almost all of our customers don’t want our load testing tool traffic to show up in their marketing statistics (Google Analytics) because it essentially renders their analytics data worthless. Additionally, Google has limits on the amount of hits per period in their free GA account; otherwise you must upgrade to the paying account. There are also legal issues involved whereby Google doesn’t want automated tools like LoadStorm hammering their applications (even GA) – it is a violation of their standard terms and conditions. Thus, when a GA tracking Javascript is found in a page that is involved in a LoadStorm test, that server is ignored automatically by our tool.

Most of our clients use a different mechanism for watching the user activity of a load test. There are many tools for monitoring the website, including several excellent options for open source and commercial products such as New Relic and Hyperic. Most of these are very inexpensive or free up to a certain level. These types of tools provide significant information regarding the user activity as well as important data on your architecture. For example, is your web server memory bound when response times start to increase drastically? Or did your database server hit 99% utilization when throughput drops?

Unfortunately not. LoadStorm is only for web applications that can be reached from the Amazon cloud. If you have an intranet application, then opening a hole in your firewall will allow LoadStorm to run load tests against it.

The short answer is: If your browser caches it, LoadStorm will cache it for a virtual user. When a virtual user completes a scenario, it is retired which causes the session to end.

Our load testing tool is designed to closely emulate the browser with a new, empty cache between sessions. So, generally your static assets will be cached for later steps in a scenario according to their header values. There are a few exceptions. We do not cache HTML resources since these are usually dynamically generated. Most LoadStorm users expect a subsequent HTML request to get this file again, thus that is how we coded it to work. If the headers are set properly, this will usually happen anyhow.

Yes, and only if you want to. LoadStorm has a checkbox on the scenario edit page to turn download images on or off. There is a similar option for controlling the download and execution of Javascript files.

Normally performance issues arise in the process of generating the dynamic page, and web servers can serve static images very quickly; however, we understand that many of our clients have websites with a large amount of media-related content, and load test results from those sites could be significantly affected by not downloading that content. Image download is especially relevant to performance if bottlenecks are in bandwidth or other network communications.

The starting and peak number of virtual users in LoadStorm represent concurrent users or the number of simulated users at a particular point in time. Each concurrent user will last for the duration of the scenario. If you have only one step in your scenario, then a concurrent user will last for less than a minute and then another one will take its place.

For example, if you have a test with one scenario having one step, starting at 10 concurrent users and peak of 10 concurrent users for 20 minutes, there will be 200-300 total users simulated (and same number of sessions). LoadStorm will create a new session for each virtual user, but it will not maintain the session after the scenario has ended, so your session should time out as you have configured it.

The LoadStorm test parameters simulate “concurrent” users not total users. The total simulated users will generally be much higher than concurrent users, but the actual count depends on lots of other factors such as the number of steps in the scenarios, pause times, etc.

Page views is a good term when reviewing Google Analytics data for traffic analysis. However, when reviewing load test data, page views can be a confusing metric. We do not track page views; rather we track requests and responses. Let’s look at the difference and how LoadStorm measures interactions with the server.

The page views per virtual user should stay consistent as long as the response time is consistent and no errors. In some tests, the response time and errors are significant and increase over the duration of the test. For example, we see some load tests that have a large number of timeouts (no response to the request in 35 seconds). That means that LoadStorm cannot move to the next request for a virtual user until the response is received or we timeout the request. This drastically changes the equation you may be using to calculate the number of pageviews. Specifically, in one test there were 143,553 errors out of 1,345,534 total requests during a 30 minute test. This failure rate greatly decreased the number of requests LoadStorm was sending during the test.

Also, if a request is sent for a step, LoadStorm is using the response header to capture the content type. A normal HTML page in a test returns, “text/html; charset=UTF-8″. When no response is received, LoadStorm assigns the response content type as, “NONE”. This affects the summary report table because the NONE responses are lumped into the OTHER category rather than the HTML category. Therefore, many of the 143k errors are shown in the OTHER category – not in the HTML category. That means that many of the HTML requests are not even shown in the table as HTML (or page views).

The bottom line is that it’s mathematically invalid to expect linear correlation between page views, requests/responses, and vusers if the target server is not responding well. Everyone has a different definition of test success and failure. Some believe that greater than 1% error rate is a reason to stop the test. Others believe that an average response time greater than 3 seconds is a failure. Regardless how you define failure, timeouts and other server errors degrade the value of your load test metrics.

The short answer is no. LoadStorm reporting graphs include a metric called Requests Per Second, and that is what you are describing.

A virtual or simulated user does what a “real” user does as specified by the scenarios and steps that you have created in LoadStorm. If there are X number of users, then there are X scenarios running at that particular time.

When creating a scenario, you can adjust the pause times which is how much time there is between steps. The actual pause is randomly generated to give a more realistic load against the server. As an example, a scenario which opens a single page for each step with an average pause of 30 seconds will have an average of 2 requests per minute per user. It gets a bit more complicated when there are external files like Javascript. These are added as well and counted in the total requests. When one scenario ends for a designated number of users, another is started within a minute to maintain the desired user load until the end of the test.

The vuser continues to move to the next step in its scenario. For instance, if a JavaScript file fails to download during step 3:

• That error will be logged in LoadStorm for reporting
• That request will be terminated
• All other requests for step 3 will be attempted as usual
• The vuser will begin step 4 as defined in the scenario

If any particular resource times out within a step, some of the step processing may not finish. This depends on what kind of resource fails and at what point in the step. If an image fails, this is relatively harmless. The images following will not be requested, but everything else will continue normally. If the html fails with a timeout, then the step will stop and no images or other resources will be requested. Images, javascript and css all depend on the html, so none will be requested for that step unless the html is received without error.

The step following a step with an error will work normally to the extent that it can. As with a browser, you cannot click on a link or submit a form if you do not currently have a valid page available with which to work. You can always type in a new url and go from there. Likewise, in LoadStorm if the next step is an open step, then previous steps will have no effect except for possibly login or cookie state. If the next step is a click or form, then the previous html must have been received for the step to function. If the step cannot work because the previous html timed out or otherwise failed, then that step will show no requests or errors and the next step will be attempted after the expected pause.

There is currently only one situation where a scenario will restart before all of its steps have run. This is a step timeout where a step starts but does not finish before 5 minutes have passed. A step includes getting all resources within the step such as html, javascript, images, css, etc. So, it is not inconceivable that this timeout occurs if there are 50, 100 or more resources in a step, but it is long enough that it is pretty rare.

LoadStorm creates the starting concurrent users within seconds of the test starting. For example, if you set the test to start at 1,000 users then you will have requests coming from those 1,000 users in the first 30 seconds. It then processes each virtual user through the scenario assigned to it. The first step is executed such as requesting your home page. The HTML is returned from your server and LoadStorm parses it into the DOM. All of the other resources are determined (images, Javascript, CSS, and XML) are requested separately and sequentially.

When those responses are all returned, then the system pauses a random time between min & max pause (you control this), then it requests the next step. That continues until the virtual user reaches the end of the scenario, whereupon that vuser is “retired”. After about 30 seconds, another vuser is created and the process starts again. The cache for the new virtual user starts empty and does not remember cookies or static resources for the previous vuser.

There is currently no functionality to add gzip to the LoadStorm request header. We hope to add that feature in an upcoming release. “Accept-Encoding: gzip, deflate” cannot be sent in the request generated by LoadStorm.

Average Duration is the time taken for ALL steps to complete (see step duration calculation), plus the average pause times between steps. Thus, it is the mean time for a specific vuser’s “life”. The actual duration fluctuates with the amount of pause assigned to that vuser. The Average Duration is shown above the steps on the Scenario page.

Duration is the time taken for a step to complete, which is the sum of time to last byte for each request in the step (including HTML, JS, CSS, images, etc.). The Average Duration shown above the steps on the Scenario page is the total of all step durations plus the average pause times between steps.

In LoadStorm the response time is calculated thus: The total time taken between the first byte of the request sent and the last byte of the response received for all requests made. This includes the network latency and the time for the server to respond.

The primary page normally includes the HTML, CSS, and Javascript files. However, the primary page could be another resource such as a PDF, video, or Flash file. The response time may appear slightly faster than observed by you in a browser because it does not include time for a browser to render the page. Of course render time will be affected by browser type, operating system, and horsepower of the client computer.

It is also important to note that LoadStorm’s servers probably have faster internet connections than many homes and offices.

LoadStorm is a truly distributed application that leverages the power of Amazon Web Services to scale on demand with processing power and bandwidth as needed to test the largest web projects. As you crank up the testing load to 200 hundred or 20,000 virtual users, LoadStorm is automatically adding machines for you (as many as necessary) from Amazon’s server farm to handle the processing. When your tests are done and the extra machines are not needed, they are turned off to wait for another test.

The normal processing is for a scenario to be processed as a sequential series of steps that you have defined in the test plan. Once a step finishes, the vuser pauses for the specified time (between min and max for scenario), then the next step proceeds. After the last step completes, the vuser context (session, cache, cookies) is cleared and the scenario starts from the beginning again with a new vuser.

You define the number of concurrent users you want, start and peak, during your test. As one vuser completes the scenario as described above, then another vuser is created to replace it – starting at step 1 again.

If you have started at a small number and set the peak number much higher, then LoadStorm will not only retire vusers that have completed the scenario, it will also add additional users throughout the test to ramp up the volume.
This is repeated until the test is done as defined by the test duration field (in minutes) when you began the load test.

If one of your web pages contains a reference to an image that is not available on your system, then a response header will be sent to LoadStorm have Status Code = 404. Additionally, your web server will send an HTML page indicating the 404 “Not Found” page to alert the user. That’s standard practice in web applications.

The reason LoadStorm reports the 404 errors as HTML is because your web server is sending our system the Content Type = “HTML” attribute. LoadStorm separates the data in the HTML and Other rows by the Content Type.

A round robin load balancer will not work very well for testing your configuration unless you are using a much higher number of virtual users (at least 2000 or so). Each of our simulation servers will resolve the DNS name independently of the others, but each of our servers will simulate hundreds of users. I would expect a peak load of 500 users to get routed to more than one of your servers, but probably not all 5 of them. A peak load of 100 users or less is likely to hit only one of your servers.

We would not expect LoadStorm test volume to show up in your Google Analytics. We do execute the javascript, but requests outside of your domain are blocked unless those other servers have also been verified. Servers like Google Analytics should be set to “Ignore”. So, the tracking by Analytics, ad servers or other systems outside of your domain would not work. Some of these kinds of services have restrictions against load testing.

Almost all of our customers don’t want our load testing tool traffic to show up in their marketing statistics (Google Analytics) because it essentially renders their analytics data worthless. Additionally, Google has limits on the amount of hits per period in their free GA account; otherwise you must upgrade to the paying account. There are also legal issues involved whereby Google doesn’t want automated tools like LoadStorm hammering their applications (even GA) – it is a violation of their standard terms and conditions. Thus, when a GA tracking Javascript is found in a page that is involved in a LoadStorm test, that server is ignored automatically by our tool.

Most of our clients use a different mechanism for watching the user activity of a load test. There are many tools for monitoring the website, including several excellent options for open source and commercial products such as New Relic and Hyperic. Most of these are very inexpensive or free up to a certain level. These types of tools provide significant information regarding the user activity as well as important data on your architecture. For example, is your web server memory bound when response times start to increase drastically? Or did your database server hit 99% utilization when throughput drops?

The short answer is: If your browser caches it, LoadStorm will cache it for a virtual user. When a virtual user completes a scenario, it is retired which causes the session to end.

Our load testing tool is designed to closely emulate the browser with a new, empty cache between sessions. So, generally your static assets will be cached for later steps in a scenario according to their header values. There are a few exceptions. We do not cache HTML resources since these are usually dynamically generated. Most LoadStorm users expect a subsequent HTML request to get this file again, thus that is how we coded it to work. If the headers are set properly, this will usually happen anyhow.

The starting and peak number of virtual users in LoadStorm represent concurrent users or the number of simulated users at a particular point in time. Each concurrent user will last for the duration of the scenario. If you have only one step in your scenario, then a concurrent user will last for less than a minute and then another one will take its place.

For example, if you have a test with one scenario having one step, starting at 10 concurrent users and peak of 10 concurrent users for 20 minutes, there will be 200-300 total users simulated (and same number of sessions). LoadStorm will create a new session for each virtual user, but it will not maintain the session after the scenario has ended, so your session should time out as you have configured it.

The LoadStorm test parameters simulate “concurrent” users not total users. The total simulated users will generally be much higher than concurrent users, but the actual count depends on lots of other factors such as the number of steps in the scenarios, pause times, etc.

Page views is a good term when reviewing Google Analytics data for traffic analysis. However, when reviewing load test data, page views can be a confusing metric. We do not track page views; rather we track requests and responses. Let’s look at the difference and how LoadStorm measures interactions with the server.

The page views per virtual user should stay consistent as long as the response time is consistent and no errors. In some tests, the response time and errors are significant and increase over the duration of the test. For example, we see some load tests that have a large number of timeouts (no response to the request in 35 seconds). That means that LoadStorm cannot move to the next request for a virtual user until the response is received or we timeout the request. This drastically changes the equation you may be using to calculate the number of pageviews. Specifically, in one test there were 143,553 errors out of 1,345,534 total requests during a 30 minute test. This failure rate greatly decreased the number of requests LoadStorm was sending during the test.

Also, if a request is sent for a step, LoadStorm is using the response header to capture the content type. A normal HTML page in a test returns, “text/html; charset=UTF-8″. When no response is received, LoadStorm assigns the response content type as, “NONE”. This affects the summary report table because the NONE responses are lumped into the OTHER category rather than the HTML category. Therefore, many of the 143k errors are shown in the OTHER category – not in the HTML category. That means that many of the HTML requests are not even shown in the table as HTML (or page views).

The bottom line is that it’s mathematically invalid to expect linear correlation between page views, requests/responses, and vusers if the target server is not responding well. Everyone has a different definition of test success and failure. Some believe that greater than 1% error rate is a reason to stop the test. Others believe that an average response time greater than 3 seconds is a failure. Regardless how you define failure, timeouts and other server errors degrade the value of your load test metrics.

The short answer is no. LoadStorm reporting graphs include a metric called Requests Per Second, and that is what you are describing.

A virtual or simulated user does what a “real” user does as specified by the scenarios and steps that you have created in LoadStorm. If there are X number of users, then there are X scenarios running at that particular time.

When creating a scenario, you can adjust the pause times which is how much time there is between steps. The actual pause is randomly generated to give a more realistic load against the server. As an example, a scenario which opens a single page for each step with an average pause of 30 seconds will have an average of 2 requests per minute per user. It gets a bit more complicated when there are external files like Javascript. These are added as well and counted in the total requests. When one scenario ends for a designated number of users, another is started within a minute to maintain the desired user load until the end of the test.

The vuser continues to move to the next step in its scenario. For instance, if a JavaScript file fails to download during step 3:

• That error will be logged in LoadStorm for reporting
• That request will be terminated
• All other requests for step 3 will be attempted as usual
• The vuser will begin step 4 as defined in the scenario

If any particular resource times out within a step, some of the step processing may not finish. This depends on what kind of resource fails and at what point in the step. If an image fails, this is relatively harmless. The images following will not be requested, but everything else will continue normally. If the html fails with a timeout, then the step will stop and no images or other resources will be requested. Images, javascript and css all depend on the html, so none will be requested for that step unless the html is received without error.

The step following a step with an error will work normally to the extent that it can. As with a browser, you cannot click on a link or submit a form if you do not currently have a valid page available with which to work. You can always type in a new url and go from there. Likewise, in LoadStorm if the next step is an open step, then previous steps will have no effect except for possibly login or cookie state. If the next step is a click or form, then the previous html must have been received for the step to function. If the step cannot work because the previous html timed out or otherwise failed, then that step will show no requests or errors and the next step will be attempted after the expected pause.

There is currently only one situation where a scenario will restart before all of its steps have run. This is a step timeout where a step starts but does not finish before 5 minutes have passed. A step includes getting all resources within the step such as html, javascript, images, css, etc. So, it is not inconceivable that this timeout occurs if there are 50, 100 or more resources in a step, but it is long enough that it is pretty rare.

LoadStorm creates the starting concurrent users within seconds of the test starting. For example, if you set the test to start at 1,000 users then you will have requests coming from those 1,000 users in the first 30 seconds. It then processes each virtual user through the scenario assigned to it. The first step is executed such as requesting your home page. The HTML is returned from your server and LoadStorm parses it into the DOM. All of the other resources are determined (images, Javascript, CSS, and XML) are requested separately and sequentially.

When those responses are all returned, then the system pauses a random time between min & max pause (you control this), then it requests the next step. That continues until the virtual user reaches the end of the scenario, whereupon that vuser is “retired”. After about 30 seconds, another vuser is created and the process starts again. The cache for the new virtual user starts empty and does not remember cookies or static resources for the previous vuser.

Our verification process works exactly like Google Analytics. The idea is to prove it is your server so LoadStorm isn’t used for a denial of service attack. You can put a file with the name generated by LS into your root directory (doesn’t matter what is in it). So let’s say your target server is http://www.xyzcorp.com, then there should be a valid response if a request comes to your server for http://www.xyzcorp.com/loadstorm-90630fc588.html.

Alternatively, if you don’t want to use the file you can embed the code in your default page. To continue the example, the HTML page received from requesting http://www.xyzcorp.com should contain a string somewhere within it (inside a comment tag is good) that matches “loadstorm-90630fc588″. LoadStorm will parse the page returned by your server exactly like a browser will. In that parsing, the verification string must appear somewhere for us to consider it verified.

Once you have the file or text in place, click on the Build Tab, then select the plan name that is targeting your server. You will see a Verify Now link in the yellow shaded section that shows your verification code. Click on that link and you should now see “Verified: Yes” on the plan page. If you have difficulty or need assistance, please contact [email protected].

Please only use the domain name when adding a server, and only use the domain name once. Adding a server for http://www.xyz.com works great and verification is successful. Then creating a server for http://www.xyz.com/myfolder will not work and trying to verify it will produce an error.

Most people create several scenarios to accurately reflect different types of users. Scenarios execute simultaneously in order to hit your site with the correct mix of traffic. You control the weighting of each scenario by giving it a number relative to the others. For example, if you have 2 scenarios and you give one a weighting of 4 and the other a weighting of 1, then the first will generate 80% of the traffic and the second 20%. This 80/20 could be realistic for an retail e-commerce site where most of the people (80%) are browsing the product catalog, while a minority of people (20%) are going through a buying experience. You can of course throw in a low-weight scenario to represent some internal employees adding products or issuing refunds.

There is no limit to the number of scenarios you can create in a test plan. We encourage you to build as many as necessary to accurately reflect the real traffic patterns on your site. That will give you the most meaningful metrics for tuning your applications.

We also simulate the user “think time” with random pauses between steps. Real users click on something, then read the page or fill in a form, and then click on something else. Thus, you don’t want to have a simulated virtual user clicking on something every .5 seconds – it would not give you meaningful results if you want to see how your site holds up in the real world.

Also see: Virtual User Calculations about calculating the amount of virtual users you need for a realistic test.

It helps you with performance engineering. We recommend that you do not begin a large test at the maximum number of virtual users. Rather than run a test for 10 minutes beginning with 5,000 and ending with 5,000, we suggest that you perhaps start with 500 users and increase to 5,000 over an hour. This approach has the advantage of allowing your system to get all the parts working properly at smaller load (e.g. caches, threads, database connections) before the heavier volume starts exposing potential bottlenecks in your application.

From what we’ve seen in thousands of load tests with LoadStorm, it is common to see useful patterns in the metrics before the peak traffic is reached, and these metrics point to areas of performance limitations that can be tuned.

One of the primary reasons we price LoadStorm subscriptions to allow unlimited test runs in the month is because we know that load testing is invariably coupled with performance tuning – an iterative process of test/tune/test/tune. Therefore, we recommend you ramp the volume in order to get more value from the test results.

Yes, there are two additional ways for you to white list our traffic:

1. The user agent in the http header includes LoadStorm. However, if your test scenarios contain steps with https requests, then those headers will be encrypted.
2. If you are not testing against your production server, we recommend trying a non-standard port number to identify the traffic as load testing. LoadStorm allows you to use any port number and can be designated for each target server.

LoadStorm follows links when the session ID is appended to the URL query string because it searches for the text between the open and close anchor tags. If it doesn’t find that text, then the fall back is to match the XPATH. We stored the XPATH when the scenario was created (essentially an address inside the page’s DOM). The XPATH is a unique identifier that allows us to click on the anchor such as an image or the href attribute or execute the Javascript.

Yes, you may submit forms such as a login form and fill-in the fields (e.g. username, password) with data from your system.

First, you export a CSV file of user information from your user table(s). The first row of the CSV must contain the names of the columns, and these column names will be mapped to the field name in the step that submits the form.

The CSV file is uploaded by clicking on the UPLOAD DATA button on the Build tab. This button will be near the bottom, under the Form Data Sets section of the page. You must give the file a label so that you can choose that file to be used at the scenario level.

If you need to embed a comma inside a particular data field, simply place quotation marks around all field data in the file.

The LoadStorm Tour has a video tutorial showing how to create a login form submission.

The Form Data Set field that you see on the Scenario settings page is a mechanism to select an input source for form submission. Using the Form Data Sets functionality, it is possible to submit forms such as a login form and fill-in the fields (e.g. username, password) with data from your system. Other common types of form submission are search terms, credit card numbers, and profile preferences. Any type of form field that can be filled in with text can utilize the Form Data Sets.

You may upload your own data to be used for form submission, or you may use one of our built-in form data sets.

The Tiny, Small, and Huge form data sets are built-into LoadStorm. The only real difference between the three files are the number of rows. Tiny only has 10 records. Small has 100, and Huge has 10,000 records of user information. They contain user information:

• First Name
• Last Name
• Username
• Password
• Email address

Many of our customers use one of our built-in form data sets for registration on their web application or for logging in virtual users.

If you are going to upload your own data into LoadStorm, here is an example of the process for getting user credentials from your system to LoadStorm:

1. Export a CSV file of user information from your user table(s).
2. The first row of the CSV must contain the names of the columns, and these column names will be mapped to the field name in the step that submits the form.
3. The CSV file is uploaded by clicking on the UPLOAD DATA button on the Build tab. This button will be near the bottom, under the Form Data Sets section of the page.
4. You must give the file a label so that you can choose that file to be used at the scenario level.

Once you have the CSV data in LoadStorm (you should see it on the Build tab under the heading Form Data Sets), then you need to edit the scenario and choose the appropriate data set based on the label you assigned when uploading it. This essentially connects the CSV file to your scenario. Then when you add a step for form submission, you will be able to map the fields in the form to the columns of data in the Form Data Set.

The LoadStorm Tour has a video tutorial showing how to create a login form submission. Watch the video on YouTube

Here are some thoughts on increasing RPS in LoadStorm tests:

1. Reducing pause time between steps (at scenario level) – minimum is 10 second pause, so set both min & max to 10.
2. Increase the number of concurrent users.
3. Reduce the response time from your server which decreases LoadStorm’s waiting for a response.

RPS will always be impacted by the types of pages/resources your scenarios are requesting. The way it works is that LoadStorm cranks up the starting concurrent users shortly after test starts. It then walks each user through the scenario assigned to it. One step is executed (such as home page), and all the images, JS, CSS, XML are requested separately. When those responses are all returned, then the system pauses a random time between min & max pause, then it requests the next step. That continues until the end of the scenario, whereupon that virtual user is retired. After a few seconds, another vuser is created and the process starts again. So, having more users going at the same time, without waiting more than a few hundred ms to get a response, will increase RPS.

We have a mechanism to “import” URLs, and we have a video tutorial that may be helpful – http://youtu.be/5APaPn2zo6M

The concept is to upload a file of strings that can be substituted in a single step of a scenario. Most customers use this functionality to append a query string on the end of a domain. You can produce a CSV with one or more columns containing the strings. Then you add a step that opens a page, and there is specific syntax for mapping the string from a column of the CSV into the URL requested by the step.

It may be helpful to review the Here’s an example of what would be used for the step:

http://cia-factbook.s3.amazonaws.com#{FileOfPaths.pathFull1}

The hash and open brace are significant because they tell LoadStorm to substitute the text. “FileOfPaths” is the label for the CSV I uploaded into my LoadStorm account. “pathFull1″ is the column name inside FileOfPaths that contains the string I want to append on the domain.

To upload the CSV (form data set) of your URLs/query strings into LoadStorm:

1. Export a CSV file with one or more columns containing the URLs.
2. The first row of the CSV must contain the names of the columns, and these column names will be mapped to the substitution point in the step indicated by the #{ } delimiters.
3. The CSV file is uploaded by clicking on the UPLOAD DATA button on the Build tab. This button will be near the bottom, under the Form Data Sets section of the page.
4. You must give the file a label so that you can choose that file to be used at the scenario level.

Once you have the CSV data in LoadStorm (you should see it on the Build tab under the heading Form Data Sets), then you need to edit the scenario and choose the appropriate data set based on the label you assigned when uploading it. This essentially connects the CSV file to your scenario. Then when you add a step for using the URL/query string, you will be able to place #{formDataSet.columnName} from the Form Data Set.

When adding a step in LoadStorm, our emulator makes the request to your web application and receives the response containing the HTML code for the page. Our tool then parses the HTML to determine what additional resources (e.g. images, Javascript libraries) need to be requested for that page.

Each of those resources are requested and received. LoadStorm saves all of the HTML and other resources in our database along with other information it needs to reproduce the sequence of events for this step during a load test. The visual rendering you see at the bottom of the step is an iFrame where our tool places a copy of what was received during the step creation process. It is NOT an accurate rendering as you would see it in a full browser without all of the other LoadStorm code and process
“wrapped around” it. Essentially, what you are seeing is a page inside a page. The images may not display because of how they are referenced (e.g. absolute or relative) in the HTML code.

The purpose of this iFrame display is to give the person building test scenarios confidence that the correct requests/responses were transacted for this step. The images you don’t see are still going to be requested during the load test, and all of the performance metrics such as response times will be accurate. Virtual users do not render the pages. No one will actually “see” the transactions during a load test because there is no need to see the pages, and our tool is measuring the Throughput, Error Rate, Response Times, Requests per Second, and Concurrent Users from many different load generation machines that process each virtual user going through the steps.

Unfortunately, our load testing tool doesn’t have a feature for what you need. Normally this is a simple process of putting a domain name to IP address mapping in a configuration file on a particular machine. The process is not as easy in our situation because we use dynamically instantiated servers from the Amazon cloud; thus, we do not know what IP addresses we will be using before you run your tests. We would need to create new images specifically for your tests, and we would need to make changes to our core launching software that instantiates the images so that new machines are launched with this particular host mapping. If you would like us to coordinate this custom process, please contact our support. Additional charges per test will apply.

Yes, you may add Storm on Demand users for any load test. This allows you to spike test traffic to a much higher level than your subscription.

For example, if you have a Hurricane subscription for 5,000 maximum concurrent vusers, you can purchase an additional 10,000 Storm on Demand users for a total of 15,000 vusers on a single test. Those 10,000 on demand users would be available for only one load test, and the 5,000 are available for other tests throughout the month.

Yes, and only if you want to. LoadStorm has a checkbox on the scenario edit page to turn download images on or off. There is a similar option for controlling the download and execution of Javascript files.

Normally performance issues arise in the process of generating the dynamic page, and web servers can serve static images very quickly; however, we understand that many of our clients have websites with a large amount of media-related content, and load test results from those sites could be significantly affected by not downloading that content. Image download is especially relevant to performance if bottlenecks are in bandwidth or other network communications.

There is currently no functionality to add gzip to the LoadStorm request header. We hope to add that feature in an upcoming release. “Accept-Encoding: gzip, deflate” cannot be sent in the request generated by LoadStorm.

LoadStorm™ is a web-based load testing tool for simulating what users do with a web site or web application. You use it to build tests that send requests to your server in the same way that a user’s browser sends requests to your server. But these tests are executed by our automated systems rather than by a user, so they can be done repeatedly and in large numbers simultaneously. They can also be built using our tool in such a way as to simulate a large number of different users with different tasks to perform.

It depends on how many simulated users you need for testing, and it depends on if you want Storm on Demand (pay per test) or Monthly Subscription.

The Breeze account is free forever (not just a trial), and it is a fully functional version that can be used for building and running any test plan or scenarios. The only limit of a free account is 25 concurrent users.

Premium subscriptions provide higher levels of load and begin at $39.90. Storm on Demand is approximately $0.0399 per virtual user.

For all the details, please see Load Testing Cost.

Subscription Highlights

• No limit to the number of tests you can run in a month.
• No obligation to subscribe beyond a month.
• Your test plans & results are preserved forever.

Our verification process works exactly like Google Analytics. The idea is to prove it is your server so LoadStorm isn’t used for a denial of service attack. You can put a file with the name generated by LS into your root directory (doesn’t matter what is in it). So let’s say your target server is http://www.xyzcorp.com, then there should be a valid response if a request comes to your server for http://www.xyzcorp.com/loadstorm-90630fc588.html.

Alternatively, if you don’t want to use the file you can embed the code in your default page. To continue the example, the HTML page received from requesting http://www.xyzcorp.com should contain a string somewhere within it (inside a comment tag is good) that matches “loadstorm-90630fc588″. LoadStorm will parse the page returned by your server exactly like a browser will. In that parsing, the verification string must appear somewhere for us to consider it verified.

Once you have the file or text in place, click on the Build Tab, then select the plan name that is targeting your server. You will see a Verify Now link in the yellow shaded section that shows your verification code. Click on that link and you should now see “Verified: Yes” on the plan page. If you have difficulty or need assistance, please contact [email protected].

Please only use the domain name when adding a server, and only use the domain name once. Adding a server for http://www.xyz.com works great and verification is successful. Then creating a server for http://www.xyz.com/myfolder will not work and trying to verify it will produce an error.

Most people create several scenarios to accurately reflect different types of users. Scenarios execute simultaneously in order to hit your site with the correct mix of traffic. You control the weighting of each scenario by giving it a number relative to the others. For example, if you have 2 scenarios and you give one a weighting of 4 and the other a weighting of 1, then the first will generate 80% of the traffic and the second 20%. This 80/20 could be realistic for an retail e-commerce site where most of the people (80%) are browsing the product catalog, while a minority of people (20%) are going through a buying experience. You can of course throw in a low-weight scenario to represent some internal employees adding products or issuing refunds.

There is no limit to the number of scenarios you can create in a test plan. We encourage you to build as many as necessary to accurately reflect the real traffic patterns on your site. That will give you the most meaningful metrics for tuning your applications.

We also simulate the user “think time” with random pauses between steps. Real users click on something, then read the page or fill in a form, and then click on something else. Thus, you don’t want to have a simulated virtual user clicking on something every .5 seconds – it would not give you meaningful results if you want to see how your site holds up in the real world.

Also see: Virtual User Calculations about calculating the amount of virtual users you need for a realistic test.

It helps you with performance engineering. We recommend that you do not begin a large test at the maximum number of virtual users. Rather than run a test for 10 minutes beginning with 5,000 and ending with 5,000, we suggest that you perhaps start with 500 users and increase to 5,000 over an hour. This approach has the advantage of allowing your system to get all the parts working properly at smaller load (e.g. caches, threads, database connections) before the heavier volume starts exposing potential bottlenecks in your application.

From what we’ve seen in thousands of load tests with LoadStorm, it is common to see useful patterns in the metrics before the peak traffic is reached, and these metrics point to areas of performance limitations that can be tuned.

One of the primary reasons we price LoadStorm subscriptions to allow unlimited test runs in the month is because we know that load testing is invariably coupled with performance tuning – an iterative process of test/tune/test/tune. Therefore, we recommend you ramp the volume in order to get more value from the test results.

LoadStorm needs a little time to prepare the test resources such as the load generation servers. Our system is using each of your scenarios and steps to pre-calculate the virtual users’ actions, how many servers will need to be instantiated from the cloud, random pauses, etc. This produces essentially a test roadmap that our Scheduler & Summarizer modules use to coordinate all the http traffic and correctly capture the metrics. Depending on the number of concurrent users and the duration of your test, the preparation of a load test can take 2 to 15 minutes.

Because we use the Amazon EC2 cloud to dynamically instantiate load generation servers, we don’t have a specific static IP address, but here are the ranges:

US East (Northern Virginia):

72.44.32.0/19 (72.44.32.0 – 72.44.63.255)
67.202.0.0/18 (67.202.0.0 – 67.202.63.255)
75.101.128.0/17 (75.101.128.0 – 75.101.255.255)
174.129.0.0/16 (174.129.0.0 – 174.129.255.255)
204.236.192.0/18 (204.236.192.0 – 204.236.255.255)
184.73.0.0/16 (184.73.0.0 – 184.73.255.255)
184.72.128.0/17 (184.72.128.0 – 184.72.255.255)
184.72.64.0/18 (184.72.64.0 – 184.72.127.255)
50.16.0.0/15 (50.16.0.0 – 50.17.255.255)
50.19.0.0/16 (50.19.0.0 – 50.19.255.255)
107.20.0.0/14 (107.20.0.0 – 107.23.255.255)
23.20.0.0/14 (23.20.0.0 – 23.23.255.255)
54.242.0.0/15 (54.242.0.0 – 54.243.255.255)
54.234.0.0/15 (54.234.0.0 – 54.235.255.255)
54.236.0.0/15 (54.236.0.0 – 54.237.255.255)
54.224.0.0/15 (54.224.0.0 – 54.225.255.255)
54.226.0.0/15 (54.226.0.0 – 54.227.255.255)
54.208.0.0/15 (54.208.0.0 – 54.209.255.255)
54.210.0.0/15 (54.210.0.0 – 54.211.255.255)
54.221.0.0/16 (54.221.0.0 – 54.221.255.255)

US West (Northern California):

204.236.128.0/18 (204.236.128.0 – 204.236.191.255)
184.72.0.0/18 (184.72.0.0 – 184.72.63.255)
50.18.0.0/16 (50.18.0.0 – 50.18.255.255)
184.169.128.0/17 (184.160.128.0 – 184.169.255.255)
54.241.0.0/16 (54.241.0.0 – 54.241.255.255)
54.215.0.0/16 (54.215.0.0 – 54.215.255.255)
54.219.0.0/16 (54.219.0.0 – 54.219.255.255)

EU (Ireland):

79.125.0.0/17 (79.125.0.0 – 79.125.127.255)
46.51.128.0/18 (46.51.128.0 – 46.51.191.255)
46.51.192.0/20 (46.51.192.0 – 46.51.207.255)
46.137.0.0/17 (46.137.0.0 – 46.137.127.255)
46.137.128.0/18 (46.137.128.0 – 46.137.191.255)
176.34.128.0/17 (176.34.128.0 – 176.34.255.255)
176.34.64.0/18 (176.34.64.0 – 176.34.127.255)
54.247.0.0/16 (54.247.0.0 – 54.247.255.255)
54.246.0.0/16 (54.246.0.0 – 54.246.255.255)
54.228.0.0/16 (54.228.0.0 – 54.228.255.255)
54.216.0.0/15 (54.216.0.0 – 54.217.255.255)
54.229.0.0/16 (54.229.0.0 – 54.229.255.255)
54.220.0.0/16 (54.220.0.0 – 54.220.255.255)

Asia Pacific (Singapore)

175.41.128.0/18 (175.41.128.0 – 175.41.191.255)
122.248.192.0/18 (122.248.192.0 – 122.248.255.255)
46.137.192.0/18 (46.137.192.0 – 46.137.255.255)
46.51.216.0/21 (46.51.216.0 – 46.51.223.255)
54.251.0.0/16 (54.251.0.0 – 54.251.255.255)
54.254.0.0/16 (54.254.0.0 – 54.254.255.255)
54.255.0.0/16 (54.255.0.0 – 54.255.255.255)

The Form Data Set field that you see on the Scenario settings page is a mechanism to select an input source for form submission. Using the Form Data Sets functionality, it is possible to submit forms such as a login form and fill-in the fields (e.g. username, password) with data from your system. Other common types of form submission are search terms, credit card numbers, and profile preferences. Any type of form field that can be filled in with text can utilize the Form Data Sets.

You may upload your own data to be used for form submission, or you may use one of our built-in form data sets.

The Tiny, Small, and Huge form data sets are built-into LoadStorm. The only real difference between the three files are the number of rows. Tiny only has 10 records. Small has 100, and Huge has 10,000 records of user information. They contain user information:

• First Name
• Last Name
• Username
• Password
• Email address

Many of our customers use one of our built-in form data sets for registration on their web application or for logging in virtual users.

If you are going to upload your own data into LoadStorm, here is an example of the process for getting user credentials from your system to LoadStorm:

1. Export a CSV file of user information from your user table(s).
2. The first row of the CSV must contain the names of the columns, and these column names will be mapped to the field name in the step that submits the form.
3. The CSV file is uploaded by clicking on the UPLOAD DATA button on the Build tab. This button will be near the bottom, under the Form Data Sets section of the page.
4. You must give the file a label so that you can choose that file to be used at the scenario level.

Once you have the CSV data in LoadStorm (you should see it on the Build tab under the heading Form Data Sets), then you need to edit the scenario and choose the appropriate data set based on the label you assigned when uploading it. This essentially connects the CSV file to your scenario. Then when you add a step for form submission, you will be able to map the fields in the form to the columns of data in the Form Data Set.

The LoadStorm Tour has a video tutorial showing how to create a login form submission. Watch the video on YouTube

When adding a step in LoadStorm, our emulator makes the request to your web application and receives the response containing the HTML code for the page. Our tool then parses the HTML to determine what additional resources (e.g. images, Javascript libraries) need to be requested for that page.

Each of those resources are requested and received. LoadStorm saves all of the HTML and other resources in our database along with other information it needs to reproduce the sequence of events for this step during a load test. The visual rendering you see at the bottom of the step is an iFrame where our tool places a copy of what was received during the step creation process. It is NOT an accurate rendering as you would see it in a full browser without all of the other LoadStorm code and process
“wrapped around” it. Essentially, what you are seeing is a page inside a page. The images may not display because of how they are referenced (e.g. absolute or relative) in the HTML code.

The purpose of this iFrame display is to give the person building test scenarios confidence that the correct requests/responses were transacted for this step. The images you don’t see are still going to be requested during the load test, and all of the performance metrics such as response times will be accurate. Virtual users do not render the pages. No one will actually “see” the transactions during a load test because there is no need to see the pages, and our tool is measuring the Throughput, Error Rate, Response Times, Requests per Second, and Concurrent Users from many different load generation machines that process each virtual user going through the steps.

Occasionally, a test will be terminated early because the target server fails or the tester stops it. It is not uncommon to have a test automatically stopped by LoadStorm when error rates hit 70% or so. When this happens in a test using Storm on Demand virtual users, those users may seem “wasted”.

Customers have asked us why the system deducted 5,000 vusers from their account when the test only lasted 10 minutes and stopped at 500 users. The reason LoadStorm does not prorate the Storm on Demand vusers to the minute or to the actual peak users is because we incur the full costs of the 5,000 vuser test.

Amazon’s cloud is elastic, and we “rent” the servers needed for your tests. However, there is a minimum of 1 hour increments that Amazon will bill us. For example, if your test needs 20 servers to hit your defined peak users, we buy those 20 servers upfront before your test actually starts. Then if you load test dies 5 minutes into it, we are still charged as if the test used all 20 servers for 1 hour.

Thus, we recommend that you start with smaller tests to verify that you have all of the environmental factors properly configured. Growing the volume in several tests that increase in volume will potentially eliminate the “wasted” feeling of throwing 5,000 users at a server that fails at 500 users.

For some projects, this is an acceptable approach. For some internal applications, apps that are very simple or realistically will expect light traffic, the best plan may be to get it done as quickly as possible. You have to ask yourself what is the risk of the application not working at some point. For many businesses, getting a large influx of new users is the peak of success, and that success would be shattered by having the new users unable to use the system, unable to buy your product and unhappy with your company.

Yes, you may add Storm on Demand users for any load test. This allows you to spike test traffic to a much higher level than your subscription.

For example, if you have a Hurricane subscription for 5,000 maximum concurrent vusers, you can purchase an additional 10,000 Storm on Demand users for a total of 15,000 vusers on a single test. Those 10,000 on demand users would be available for only one load test, and the 5,000 are available for other tests throughout the month.

Yes, and only if you want to. LoadStorm has a checkbox on the scenario edit page to turn download images on or off. There is a similar option for controlling the download and execution of Javascript files.

Normally performance issues arise in the process of generating the dynamic page, and web servers can serve static images very quickly; however, we understand that many of our clients have websites with a large amount of media-related content, and load test results from those sites could be significantly affected by not downloading that content. Image download is especially relevant to performance if bottlenecks are in bandwidth or other network communications.

It depends on how many simulated users you need for testing, and it depends on if you want Storm on Demand (pay per test) or Monthly Subscription.

The Breeze account is free forever (not just a trial), and it is a fully functional version that can be used for building and running any test plan or scenarios. The only limit of a free account is 25 concurrent users.

Premium subscriptions provide higher levels of load and begin at $39.90. Storm on Demand is approximately $0.0399 per virtual user.

For all the details, please see Load Testing Cost.

Subscription Highlights

• No limit to the number of tests you can run in a month.
• No obligation to subscribe beyond a month.
• Your test plans & results are preserved forever.

LoadStorm emulates a browser. If we used an actual browser, we would trade-off the cost effectiveness of our solution for the ability to actually process exactly as a browser. Some solutions use actual browsers and cost you about 200-1,000% more per Virtual User Hour of testing.

We understand some web developers need the real browsers and are willing to pay for it. Our load testing tool is designed to help those developers that achieve their load testing goals with emulation and a much lower cost.

LoadStorm’s emulator makes the requests just like a browser, and it parses the HTML received in the response. As it parses the page, it identifies the additional resources needed such as images, stylesheets, XML, and Javascript library files. LoadStorm will automatically send requests to your server for these files. Each response is measured and recorded for speed and status codes.

Our emulator will respect caching. For example, if you have the same logo image on every page, LoadStorm will only request it once for each virtual user, and subsequent steps will request any new images but not the ones already returned by your server.

LoadStorm will process the Javascript that is triggered on page load. However, our emulator will not process background requests of Javascript or Ajax. We acknowledge the inability to process Ajax is a limitation of our tool, and we recognize that many Rich Internet Applications cannot use our tool for testing. We are working on a new solution that will remove this limitation.

Occasionally, a test will be terminated early because the target server fails or the tester stops it. It is not uncommon to have a test automatically stopped by LoadStorm when error rates hit 70% or so. When this happens in a test using Storm on Demand virtual users, those users may seem “wasted”.

Customers have asked us why the system deducted 5,000 vusers from their account when the test only lasted 10 minutes and stopped at 500 users. The reason LoadStorm does not prorate the Storm on Demand vusers to the minute or to the actual peak users is because we incur the full costs of the 5,000 vuser test.

Amazon’s cloud is elastic, and we “rent” the servers needed for your tests. However, there is a minimum of 1 hour increments that Amazon will bill us. For example, if your test needs 20 servers to hit your defined peak users, we buy those 20 servers upfront before your test actually starts. Then if you load test dies 5 minutes into it, we are still charged as if the test used all 20 servers for 1 hour.

Thus, we recommend that you start with smaller tests to verify that you have all of the environmental factors properly configured. Growing the volume in several tests that increase in volume will potentially eliminate the “wasted” feeling of throwing 5,000 users at a server that fails at 500 users.

Our support team receives this question frequently. In fact, a good example is the email this week from a new user stating, “Just tried to test your service and it came back with these comments:

I am not sure what to do now…Can you help me?”

Yes, we can help with the verification process if you are stuck.  Please see our knowledgebase for how to ‎Verify a Server.

Are You Load Testing Facebook?

However, the error messages above indicate that his test plan will be hitting servers that aren’t verified (authorized for testing). Put another way, his scenarios contain requests that go to servers that aren’t his, and LoadStorm gets in trouble when our customers load test other people’s servers without their permission.

Does he really want to hit Twitter and Facebook for his load test? If so, is he testing their ability to handle his load? If so, does he have their permission to hammer on their servers?  Probably not. They tend to get upset when someone utilizes a cloud load testing tool to simulate thousands of VUsers against their social media platform.  Please read the terms and conditions of use for those services.

And where do those VUsers originate?  LoadStorm.  So who gets the phone call from their lawyers?  We do.

Prevent Denial of Service Attacks

Our verification process is designed to prevent LoadStorm from being used as a Denial of Service (DOS) weapon.  By placing a code on your home page that LoadStorm can request and confirm, our system confirms that you have the control over that server; thus, it is safe for our tool to hammer it with VUsers.

Another option is to Ignore the server.  That simply means that LoadStorm will not send requests to it.  By ignoring a server, your test may continue and generate load against your servers without creating a DOS against Facebook or Twitter.

It helps you with performance engineering. We recommend that you do not begin a large test at the maximum number of virtual users. Rather than run a test for 10 minutes beginning with 5,000 and ending with 5,000, we suggest that you perhaps start with 500 users and increase to 5,000 over an hour. This approach has the advantage of allowing your system to get all the parts working properly at smaller load (e.g. caches, threads, database connections) before the heavier volume starts exposing potential bottlenecks in your application.

From what we’ve seen in thousands of load tests with LoadStorm, it is common to see useful patterns in the metrics before the peak traffic is reached, and these metrics point to areas of performance limitations that can be tuned.

One of the primary reasons we price LoadStorm subscriptions to allow unlimited test runs in the month is because we know that load testing is invariably coupled with performance tuning – an iterative process of test/tune/test/tune. Therefore, we recommend you ramp the volume in order to get more value from the test results.

LoadStorm needs a little time to prepare the test resources such as the load generation servers. Our system is using each of your scenarios and steps to pre-calculate the virtual users’ actions, how many servers will need to be instantiated from the cloud, random pauses, etc. This produces essentially a test roadmap that our Scheduler & Summarizer modules use to coordinate all the http traffic and correctly capture the metrics. Depending on the number of concurrent users and the duration of your test, the preparation of a load test can take 2 to 15 minutes.

LoadStorm follows links when the session ID is appended to the URL query string because it searches for the text between the open and close anchor tags. If it doesn’t find that text, then the fall back is to match the XPATH. We stored the XPATH when the scenario was created (essentially an address inside the page’s DOM). The XPATH is a unique identifier that allows us to click on the anchor such as an image or the href attribute or execute the Javascript.

LoadStorm is a truly distributed application that leverages the power of Amazon Web Services to scale on demand with processing power and bandwidth as needed to test the largest web projects. As you crank up the testing load to 200 hundred or 20,000 virtual users, LoadStorm is automatically adding machines for you (as many as necessary) from Amazon’s server farm to handle the processing. When your tests are done and the extra machines are not needed, they are turned off to wait for another test.

The normal processing is for a scenario to be processed as a sequential series of steps that you have defined in the test plan. Once a step finishes, the vuser pauses for the specified time (between min and max for scenario), then the next step proceeds. After the last step completes, the vuser context (session, cache, cookies) is cleared and the scenario starts from the beginning again with a new vuser.

You define the number of concurrent users you want, start and peak, during your test. As one vuser completes the scenario as described above, then another vuser is created to replace it – starting at step 1 again.

If you have started at a small number and set the peak number much higher, then LoadStorm will not only retire vusers that have completed the scenario, it will also add additional users throughout the test to ramp up the volume.
This is repeated until the test is done as defined by the test duration field (in minutes) when you began the load test.

Here are some thoughts on increasing RPS in LoadStorm tests:

1. Reducing pause time between steps (at scenario level) – minimum is 10 second pause, so set both min & max to 10.
2. Increase the number of concurrent users.
3. Reduce the response time from your server which decreases LoadStorm’s waiting for a response.

RPS will always be impacted by the types of pages/resources your scenarios are requesting. The way it works is that LoadStorm cranks up the starting concurrent users shortly after test starts. It then walks each user through the scenario assigned to it. One step is executed (such as home page), and all the images, JS, CSS, XML are requested separately. When those responses are all returned, then the system pauses a random time between min & max pause, then it requests the next step. That continues until the end of the scenario, whereupon that virtual user is retired. After a few seconds, another vuser is created and the process starts again. So, having more users going at the same time, without waiting more than a few hundred ms to get a response, will increase RPS.

Occasionally, a test will be terminated early because the target server fails or the tester stops it. It is not uncommon to have a test automatically stopped by LoadStorm when error rates hit 70% or so. When this happens in a test using Storm on Demand virtual users, those users may seem “wasted”.

Customers have asked us why the system deducted 5,000 vusers from their account when the test only lasted 10 minutes and stopped at 500 users. The reason LoadStorm does not prorate the Storm on Demand vusers to the minute or to the actual peak users is because we incur the full costs of the 5,000 vuser test.

Amazon’s cloud is elastic, and we “rent” the servers needed for your tests. However, there is a minimum of 1 hour increments that Amazon will bill us. For example, if your test needs 20 servers to hit your defined peak users, we buy those 20 servers upfront before your test actually starts. Then if you load test dies 5 minutes into it, we are still charged as if the test used all 20 servers for 1 hour.

Thus, we recommend that you start with smaller tests to verify that you have all of the environmental factors properly configured. Growing the volume in several tests that increase in volume will potentially eliminate the “wasted” feeling of throwing 5,000 users at a server that fails at 500 users.

You can press the “STOP NOW” button at any time. It is found on the Run tab, and the button will only appear during the execution of a load test.

Also, LoadStorm will automatically stop a load test if it can determine that your server has been overloaded. The thresholds for stopping a test are currently when the Error Rate exceeds 50% or the Average Response Time exceeds 15 seconds for the most recently summarized test period.

Important Note for Small Tests: The sample interval is not the entire test but just the most recent part of it, and there must be a minimum number of requests in the sample. This interval is significant because in very small tests the number of failed requests AND the total requests may be few. For example, you must have some requests for at least 2 intervals, which means you have summarized more than 1 minute of requests.

A round robin load balancer will not work very well for testing your configuration unless you are using a much higher number of virtual users (at least 2000 or so). Each of our simulation servers will resolve the DNS name independently of the others, but each of our servers will simulate hundreds of users. I would expect a peak load of 500 users to get routed to more than one of your servers, but probably not all 5 of them. A peak load of 100 users or less is likely to hit only one of your servers.

We would not expect LoadStorm test volume to show up in your Google Analytics. We do execute the javascript, but requests outside of your domain are blocked unless those other servers have also been verified. Servers like Google Analytics should be set to “Ignore”. So, the tracking by Analytics, ad servers or other systems outside of your domain would not work. Some of these kinds of services have restrictions against load testing.

Almost all of our customers don’t want our load testing tool traffic to show up in their marketing statistics (Google Analytics) because it essentially renders their analytics data worthless. Additionally, Google has limits on the amount of hits per period in their free GA account; otherwise you must upgrade to the paying account. There are also legal issues involved whereby Google doesn’t want automated tools like LoadStorm hammering their applications (even GA) – it is a violation of their standard terms and conditions. Thus, when a GA tracking Javascript is found in a page that is involved in a LoadStorm test, that server is ignored automatically by our tool.

Most of our clients use a different mechanism for watching the user activity of a load test. There are many tools for monitoring the website, including several excellent options for open source and commercial products such as New Relic and Hyperic. Most of these are very inexpensive or free up to a certain level. These types of tools provide significant information regarding the user activity as well as important data on your architecture. For example, is your web server memory bound when response times start to increase drastically? Or did your database server hit 99% utilization when throughput drops?

Unfortunately not. LoadStorm is only for web applications that can be reached from the Amazon cloud. If you have an intranet application, then opening a hole in your firewall will allow LoadStorm to run load tests against it.

The short answer is: If your browser caches it, LoadStorm will cache it for a virtual user. When a virtual user completes a scenario, it is retired which causes the session to end.

Our load testing tool is designed to closely emulate the browser with a new, empty cache between sessions. So, generally your static assets will be cached for later steps in a scenario according to their header values. There are a few exceptions. We do not cache HTML resources since these are usually dynamically generated. Most LoadStorm users expect a subsequent HTML request to get this file again, thus that is how we coded it to work. If the headers are set properly, this will usually happen anyhow.

The vuser continues to move to the next step in its scenario. For instance, if a JavaScript file fails to download during step 3:

• That error will be logged in LoadStorm for reporting
• That request will be terminated
• All other requests for step 3 will be attempted as usual
• The vuser will begin step 4 as defined in the scenario

If any particular resource times out within a step, some of the step processing may not finish. This depends on what kind of resource fails and at what point in the step. If an image fails, this is relatively harmless. The images following will not be requested, but everything else will continue normally. If the html fails with a timeout, then the step will stop and no images or other resources will be requested. Images, javascript and css all depend on the html, so none will be requested for that step unless the html is received without error.

The step following a step with an error will work normally to the extent that it can. As with a browser, you cannot click on a link or submit a form if you do not currently have a valid page available with which to work. You can always type in a new url and go from there. Likewise, in LoadStorm if the next step is an open step, then previous steps will have no effect except for possibly login or cookie state. If the next step is a click or form, then the previous html must have been received for the step to function. If the step cannot work because the previous html timed out or otherwise failed, then that step will show no requests or errors and the next step will be attempted after the expected pause.

There is currently only one situation where a scenario will restart before all of its steps have run. This is a step timeout where a step starts but does not finish before 5 minutes have passed. A step includes getting all resources within the step such as html, javascript, images, css, etc. So, it is not inconceivable that this timeout occurs if there are 50, 100 or more resources in a step, but it is long enough that it is pretty rare.

LoadStorm creates the starting concurrent users within seconds of the test starting. For example, if you set the test to start at 1,000 users then you will have requests coming from those 1,000 users in the first 30 seconds. It then processes each virtual user through the scenario assigned to it. The first step is executed such as requesting your home page. The HTML is returned from your server and LoadStorm parses it into the DOM. All of the other resources are determined (images, Javascript, CSS, and XML) are requested separately and sequentially.

When those responses are all returned, then the system pauses a random time between min & max pause (you control this), then it requests the next step. That continues until the virtual user reaches the end of the scenario, whereupon that vuser is “retired”. After about 30 seconds, another vuser is created and the process starts again. The cache for the new virtual user starts empty and does not remember cookies or static resources for the previous vuser.

Our support team receives this question frequently. In fact, a good example is the email this week from a new user stating, “Just tried to test your service and it came back with these comments:

I am not sure what to do now…Can you help me?”

Yes, we can help with the verification process if you are stuck.  Please see our knowledgebase for how to ‎Verify a Server.

Are You Load Testing Facebook?

However, the error messages above indicate that his test plan will be hitting servers that aren’t verified (authorized for testing). Put another way, his scenarios contain requests that go to servers that aren’t his, and LoadStorm gets in trouble when our customers load test other people’s servers without their permission.

Does he really want to hit Twitter and Facebook for his load test? If so, is he testing their ability to handle his load? If so, does he have their permission to hammer on their servers?  Probably not. They tend to get upset when someone utilizes a cloud load testing tool to simulate thousands of VUsers against their social media platform.  Please read the terms and conditions of use for those services.

And where do those VUsers originate?  LoadStorm.  So who gets the phone call from their lawyers?  We do.

Prevent Denial of Service Attacks

Our verification process is designed to prevent LoadStorm from being used as a Denial of Service (DOS) weapon.  By placing a code on your home page that LoadStorm can request and confirm, our system confirms that you have the control over that server; thus, it is safe for our tool to hammer it with VUsers.

Another option is to Ignore the server.  That simply means that LoadStorm will not send requests to it.  By ignoring a server, your test may continue and generate load against your servers without creating a DOS against Facebook or Twitter.

Our verification process works exactly like Google Analytics. The idea is to prove it is your server so LoadStorm isn’t used for a denial of service attack. You can put a file with the name generated by LS into your root directory (doesn’t matter what is in it). So let’s say your target server is http://www.xyzcorp.com, then there should be a valid response if a request comes to your server for http://www.xyzcorp.com/loadstorm-90630fc588.html.

Alternatively, if you don’t want to use the file you can embed the code in your default page. To continue the example, the HTML page received from requesting http://www.xyzcorp.com should contain a string somewhere within it (inside a comment tag is good) that matches “loadstorm-90630fc588″. LoadStorm will parse the page returned by your server exactly like a browser will. In that parsing, the verification string must appear somewhere for us to consider it verified.

Once you have the file or text in place, click on the Build Tab, then select the plan name that is targeting your server. You will see a Verify Now link in the yellow shaded section that shows your verification code. Click on that link and you should now see “Verified: Yes” on the plan page. If you have difficulty or need assistance, please contact [email protected].

Please only use the domain name when adding a server, and only use the domain name once. Adding a server for http://www.xyz.com works great and verification is successful. Then creating a server for http://www.xyz.com/myfolder will not work and trying to verify it will produce an error.

LoadStorm emulates a browser. If we used an actual browser, we would trade-off the cost effectiveness of our solution for the ability to actually process exactly as a browser. Some solutions use actual browsers and cost you about 200-1,000% more per Virtual User Hour of testing.

We understand some web developers need the real browsers and are willing to pay for it. Our load testing tool is designed to help those developers that achieve their load testing goals with emulation and a much lower cost.

LoadStorm’s emulator makes the requests just like a browser, and it parses the HTML received in the response. As it parses the page, it identifies the additional resources needed such as images, stylesheets, XML, and Javascript library files. LoadStorm will automatically send requests to your server for these files. Each response is measured and recorded for speed and status codes.

Our emulator will respect caching. For example, if you have the same logo image on every page, LoadStorm will only request it once for each virtual user, and subsequent steps will request any new images but not the ones already returned by your server.

LoadStorm will process the Javascript that is triggered on page load. However, our emulator will not process background requests of Javascript or Ajax. We acknowledge the inability to process Ajax is a limitation of our tool, and we recognize that many Rich Internet Applications cannot use our tool for testing. We are working on a new solution that will remove this limitation.

Because we use the Amazon EC2 cloud to dynamically instantiate load generation servers, we don’t have a specific static IP address, but here are the ranges:

US East (Northern Virginia):

72.44.32.0/19 (72.44.32.0 – 72.44.63.255)
67.202.0.0/18 (67.202.0.0 – 67.202.63.255)
75.101.128.0/17 (75.101.128.0 – 75.101.255.255)
174.129.0.0/16 (174.129.0.0 – 174.129.255.255)
204.236.192.0/18 (204.236.192.0 – 204.236.255.255)
184.73.0.0/16 (184.73.0.0 – 184.73.255.255)
184.72.128.0/17 (184.72.128.0 – 184.72.255.255)
184.72.64.0/18 (184.72.64.0 – 184.72.127.255)
50.16.0.0/15 (50.16.0.0 – 50.17.255.255)
50.19.0.0/16 (50.19.0.0 – 50.19.255.255)
107.20.0.0/14 (107.20.0.0 – 107.23.255.255)
23.20.0.0/14 (23.20.0.0 – 23.23.255.255)
54.242.0.0/15 (54.242.0.0 – 54.243.255.255)
54.234.0.0/15 (54.234.0.0 – 54.235.255.255)
54.236.0.0/15 (54.236.0.0 – 54.237.255.255)
54.224.0.0/15 (54.224.0.0 – 54.225.255.255)
54.226.0.0/15 (54.226.0.0 – 54.227.255.255)
54.208.0.0/15 (54.208.0.0 – 54.209.255.255)
54.210.0.0/15 (54.210.0.0 – 54.211.255.255)
54.221.0.0/16 (54.221.0.0 – 54.221.255.255)

US West (Northern California):

204.236.128.0/18 (204.236.128.0 – 204.236.191.255)
184.72.0.0/18 (184.72.0.0 – 184.72.63.255)
50.18.0.0/16 (50.18.0.0 – 50.18.255.255)
184.169.128.0/17 (184.160.128.0 – 184.169.255.255)
54.241.0.0/16 (54.241.0.0 – 54.241.255.255)
54.215.0.0/16 (54.215.0.0 – 54.215.255.255)
54.219.0.0/16 (54.219.0.0 – 54.219.255.255)

EU (Ireland):

79.125.0.0/17 (79.125.0.0 – 79.125.127.255)
46.51.128.0/18 (46.51.128.0 – 46.51.191.255)
46.51.192.0/20 (46.51.192.0 – 46.51.207.255)
46.137.0.0/17 (46.137.0.0 – 46.137.127.255)
46.137.128.0/18 (46.137.128.0 – 46.137.191.255)
176.34.128.0/17 (176.34.128.0 – 176.34.255.255)
176.34.64.0/18 (176.34.64.0 – 176.34.127.255)
54.247.0.0/16 (54.247.0.0 – 54.247.255.255)
54.246.0.0/16 (54.246.0.0 – 54.246.255.255)
54.228.0.0/16 (54.228.0.0 – 54.228.255.255)
54.216.0.0/15 (54.216.0.0 – 54.217.255.255)
54.229.0.0/16 (54.229.0.0 – 54.229.255.255)
54.220.0.0/16 (54.220.0.0 – 54.220.255.255)

Asia Pacific (Singapore)

175.41.128.0/18 (175.41.128.0 – 175.41.191.255)
122.248.192.0/18 (122.248.192.0 – 122.248.255.255)
46.137.192.0/18 (46.137.192.0 – 46.137.255.255)
46.51.216.0/21 (46.51.216.0 – 46.51.223.255)
54.251.0.0/16 (54.251.0.0 – 54.251.255.255)
54.254.0.0/16 (54.254.0.0 – 54.254.255.255)
54.255.0.0/16 (54.255.0.0 – 54.255.255.255)

Yes, there are two additional ways for you to white list our traffic:

1. The user agent in the http header includes LoadStorm. However, if your test scenarios contain steps with https requests, then those headers will be encrypted.
2. If you are not testing against your production server, we recommend trying a non-standard port number to identify the traffic as load testing. LoadStorm allows you to use any port number and can be designated for each target server.

Yes, you may submit forms such as a login form and fill-in the fields (e.g. username, password) with data from your system.

First, you export a CSV file of user information from your user table(s). The first row of the CSV must contain the names of the columns, and these column names will be mapped to the field name in the step that submits the form.

The CSV file is uploaded by clicking on the UPLOAD DATA button on the Build tab. This button will be near the bottom, under the Form Data Sets section of the page. You must give the file a label so that you can choose that file to be used at the scenario level.

If you need to embed a comma inside a particular data field, simply place quotation marks around all field data in the file.

The LoadStorm Tour has a video tutorial showing how to create a login form submission.

When adding a step in LoadStorm, our emulator makes the request to your web application and receives the response containing the HTML code for the page. Our tool then parses the HTML to determine what additional resources (e.g. images, Javascript libraries) need to be requested for that page.

Each of those resources are requested and received. LoadStorm saves all of the HTML and other resources in our database along with other information it needs to reproduce the sequence of events for this step during a load test. The visual rendering you see at the bottom of the step is an iFrame where our tool places a copy of what was received during the step creation process. It is NOT an accurate rendering as you would see it in a full browser without all of the other LoadStorm code and process
“wrapped around” it. Essentially, what you are seeing is a page inside a page. The images may not display because of how they are referenced (e.g. absolute or relative) in the HTML code.

The purpose of this iFrame display is to give the person building test scenarios confidence that the correct requests/responses were transacted for this step. The images you don’t see are still going to be requested during the load test, and all of the performance metrics such as response times will be accurate. Virtual users do not render the pages. No one will actually “see” the transactions during a load test because there is no need to see the pages, and our tool is measuring the Throughput, Error Rate, Response Times, Requests per Second, and Concurrent Users from many different load generation machines that process each virtual user going through the steps.

You can press the “STOP NOW” button at any time. It is found on the Run tab, and the button will only appear during the execution of a load test.

Also, LoadStorm will automatically stop a load test if it can determine that your server has been overloaded. The thresholds for stopping a test are currently when the Error Rate exceeds 50% or the Average Response Time exceeds 15 seconds for the most recently summarized test period.

Important Note for Small Tests: The sample interval is not the entire test but just the most recent part of it, and there must be a minimum number of requests in the sample. This interval is significant because in very small tests the number of failed requests AND the total requests may be few. For example, you must have some requests for at least 2 intervals, which means you have summarized more than 1 minute of requests.

Unfortunately not. LoadStorm is only for web applications that can be reached from the Amazon cloud. If you have an intranet application, then opening a hole in your firewall will allow LoadStorm to run load tests against it.

The vuser continues to move to the next step in its scenario. For instance, if a JavaScript file fails to download during step 3:

• That error will be logged in LoadStorm for reporting
• That request will be terminated
• All other requests for step 3 will be attempted as usual
• The vuser will begin step 4 as defined in the scenario

If any particular resource times out within a step, some of the step processing may not finish. This depends on what kind of resource fails and at what point in the step. If an image fails, this is relatively harmless. The images following will not be requested, but everything else will continue normally. If the html fails with a timeout, then the step will stop and no images or other resources will be requested. Images, javascript and css all depend on the html, so none will be requested for that step unless the html is received without error.

The step following a step with an error will work normally to the extent that it can. As with a browser, you cannot click on a link or submit a form if you do not currently have a valid page available with which to work. You can always type in a new url and go from there. Likewise, in LoadStorm if the next step is an open step, then previous steps will have no effect except for possibly login or cookie state. If the next step is a click or form, then the previous html must have been received for the step to function. If the step cannot work because the previous html timed out or otherwise failed, then that step will show no requests or errors and the next step will be attempted after the expected pause.

There is currently only one situation where a scenario will restart before all of its steps have run. This is a step timeout where a step starts but does not finish before 5 minutes have passed. A step includes getting all resources within the step such as html, javascript, images, css, etc. So, it is not inconceivable that this timeout occurs if there are 50, 100 or more resources in a step, but it is long enough that it is pretty rare.