All FAQ

  • I can’t run a load test because a server isn’t verified or ignored. Can you help me?

    Manage Servers

    We can help with the verification process if you are stuck.  Please see our learning center for how to ‎Verify a Server.

    Another option is to Ignore the server. That simply means that LoadStorm will not send requests to it. By ignoring a server, your test may continue and generate load against your servers without making unnecessary requests to external sites such as Facebook, Twitter, or even Google Analytics.

    How does verification work?

    Our verification process works exactly like Google Analytics. The idea is to prove it is your server so LoadStorm isn’t used for a denial-of-service attack. You can put an empty file that is named with a unique string generated by LoadStorm (i.e. loadstorm-<unique string>) into your root directory (doesn’t matter what is in it). So let’s say your target server is http://www.xyzcorp.com, then there should be a valid response if a request comes to your server for http://www.xyzcorp.com/loadstorm-90630fc588.html.

    Alternatively, if you don’t want to use the file you can embed the code in your default page using comment tags. To continue the example, the HTML page received from requesting http://www.xyzcorp.com should contain a string within comment tags somewhere inside the document that matches “loadstorm-90630fc588″. LoadStorm will parse the page returned by your server exactly like a browser will. In that parsing, the verification string must appear somewhere for us to consider it verified.

    Example Code with a string in a comment tag:

    <html>
    <!– loadstorm-90630fc588 –>
    </html>

    Once you have the file or text in place, click on the Build… Servers tab, then select the the domain from the table that is your target server. Click the Check/Verify button under the verification section. If you are still having difficulties or need extra assistance, please contact support@loadstorm.com.

    Please only use the domain name when adding a server, and only use the domain name once. Adding a server for http://www.xyz.com works great and verification is successful. However, creating a server for http://www.xyz.com/myfolder will not work and trying to verify it will produce an error.

    Wonder why you need to verify servers?

    Please see our discussion on why servers need to be verified..

  • What Does LoadStorm Do?

    load testing infoLoadStorm™ is a web-based load testing tool for simulating what users do with a web site or web application. You use it to build tests that send requests to your server in the same way that a user’s browser sends requests to your server. But these tests are executed by our automated systems rather than by a user, so they can be done repeatedly and in large numbers simultaneously. They can also be built using our tool in such a way as to simulate a large number of different users with different tasks to perform.

     

  • How much does it cost?

    load testing costIt depends on how many simulated users you need for testing, and it depends on if you would like to use our consulting services.

    Creating an account is free, but the only limit of a free account is 50 concurrent users.

    For more details, please see Load Testing Cost.

  • Why do I need to verify my server? How do I do that?

    This is a very common question. In fact, a good example is the email this week from a new user stating, “Just tried to test your service and it came back with these comments:

    There were problems with the following fields:

    I am not sure what to do now…Why do I have to verify these servers?”

    Are You Load Testing Facebook?

    However, the error messages above indicate that his test plan will be hitting servers that aren’t verified (authorized for testing). Put another way, his scenarios contain requests that go to servers that aren’t his, and LoadStorm gets in trouble when our customers load test other people’s servers without their permission.

    Does he really want to hit Twitter and Facebook for his load test? If so, is he testing their ability to handle his load? If so, does he have their permission to hammer on their servers? Probably not. They tend to get upset when someone utilizes a cloud load testing tool to simulate thousands of VUsers against their social media platform. Please read the terms and conditions of use for those services.

    And where do those VUsers originate? LoadStorm. So who gets the phone call from their lawyers? We do.

    Prevent Denial of Service Attacks

    Our verification process is designed to prevent LoadStorm from being used as a Denial of Service (DOS) weapon. By placing a code on your home page that LoadStorm can request and confirm, our system confirms that you have the control over that server; thus, it is safe for our tool to hammer it with VUsers.

     

  • How should I set up scripts to simulate real traffic?

    Most people create several scripts to accurately reflect different types of users. Scripts execute simultaneously in order to hit your site with the correct mix of traffic. You control the weighting of each script by giving it a number relative to the others. For example, if you have 2 scripts and you give one a weighting of 4 and the other a weighting of 1, then the first will generate 80% of the traffic and the second 20%. This 80/20 could be realistic for a retail e-commerce site where most of the people (80%) are browsing the product catalog, while a minority of people (20%) are going through a buying experience. You can of course throw in a low-weight scenario to represent some internal employees adding products or issuing refunds.

    Another way of entering the weight of the traffic is to just enter the exact percentage you would like each script to carry. For the example about you would simply enter 80 and 20. The only thing you have to be very careful about is that the percentages add up to 100. Otherwise, it will work like the above description, which is different.

    There is no limit to the number of scripts you can create in a test plan. We encourage you to build as many as necessary to accurately reflect the real traffic patterns on your site. That will give you the most meaningful metrics for tuning your applications.

    We also simulate the user “think time” with random pauses between pages. Real users click on something, then read the page or fill in a form, and then click on something else. Thus, you don’t want to have a simulated virtual user clicking on something every .5 seconds – it would not give you meaningful results if you want to see how your site holds up in the real world.

    Also see: Virtual User Calculations about calculating the amount of virtual users you need for a realistic test.

     

  • Why should I ramp up my test load volume over time?

    It helps you with performance engineering. We recommend that you do not begin a large test at the maximum number of virtual users. Rather than run a test for 10 minutes beginning with 5,000 and ending with 5,000, we suggest that you perhaps start with 500 users and increase to 5,000 over an hour. This approach has the advantage of allowing your system to get all the parts working properly at smaller load (e.g. caches, threads, database connections) before the heavier volume starts exposing potential bottlenecks in your application.

    From what we’ve seen in thousands of load tests with LoadStorm, it is common to see useful patterns in the metrics before the peak traffic is reached, and these metrics point to areas of performance limitations that can be tuned.

    Load testing is invariably coupled with performance tuning – an iterative process of test/tune/test/tune. Therefore, we recommend you ramp the volume in order to get more value from the test results.

     

  • Why doesn’t my test start immediately?

    LoadStorm needs a little time to prepare the test resources such as the load generation servers. Our system is using each of your scripts and steps to pre-calculate the virtual users’ actions, how many servers will need to be instantiated from the cloud, random think times, etc. This produces essentially a test roadmap that our Scheduler and Summarizer modules use to coordinate all the HTTP traffic and correctly capture the metrics. Depending on the number of concurrent users and the duration of your test, the preparation of a load test can take 2 to 15 minutes.

     

  • Does LoadStorm use a full browser for accessing the websites to test or does it emulate a browser?

    load testing ramp timeLoadStorm emulates a browser. If we used an actual browser, we would trade-off the cost effectiveness of our solution for the ability to actually process exactly as a browser. Some solutions use actual browsers and cost you about 200-1,000% more per Virtual User Hour of testing.

    We understand some web developers need the real browsers and are willing to pay for it. Our load testing tool is designed to help those developers that achieve their load testing goals with emulation and a much lower cost.

    LoadStorm’s emulator makes the requests just like a browser, and it parses the HTML received in the response. As it parses the page, it identifies the additional resources needed such as images, stylesheets, XML, and javascript files. LoadStorm will automatically send requests to your server for these files. Each response is measured and recorded for speed and status codes.

    Our emulator will respect caching. For example, if you have the same logo image on every page, LoadStorm will only request it once for each virtual user, and subsequent steps will request any new images but not the ones already returned by your server.

    LoadStorm 2.0 effectively supports AJAX.

     

  • What are the IP address ranges used by LoadStorm? We need to white list them.

    Because we use the Amazon EC2 cloud to dynamically instantiate load generation servers, we don’t have a specific static IP address, but here are the ranges:

    US East (Northern Virginia):

    72.44.32.0/19 (72.44.32.0 – 72.44.63.255)
    67.202.0.0/18 (67.202.0.0 – 67.202.63.255)
    75.101.128.0/17 (75.101.128.0 – 75.101.255.255)
    174.129.0.0/16 (174.129.0.0 – 174.129.255.255)
    204.236.192.0/18 (204.236.192.0 – 204.236.255.255)
    184.73.0.0/16 (184.73.0.0 – 184.73.255.255)
    184.72.128.0/17 (184.72.128.0 – 184.72.255.255)
    184.72.64.0/18 (184.72.64.0 – 184.72.127.255)
    50.16.0.0/15 (50.16.0.0 – 50.17.255.255)
    50.19.0.0/16 (50.19.0.0 – 50.19.255.255)
    107.20.0.0/14 (107.20.0.0 – 107.23.255.255)
    23.20.0.0/14 (23.20.0.0 – 23.23.255.255)
    54.242.0.0/15 (54.242.0.0 – 54.243.255.255)
    54.234.0.0/15 (54.234.0.0 – 54.235.255.255)
    54.236.0.0/15 (54.236.0.0 – 54.237.255.255)
    54.224.0.0/15 (54.224.0.0 – 54.225.255.255)
    54.226.0.0/15 (54.226.0.0 – 54.227.255.255)
    54.208.0.0/15 (54.208.0.0 – 54.209.255.255)
    54.210.0.0/15 (54.210.0.0 – 54.211.255.255)
    54.221.0.0/16 (54.221.0.0 – 54.221.255.255)
    54.204.0.0/15 (54.204.0.0 – 54.205.255.255)
    54.196.0.0/15 (54.196.0.0 – 54.197.255.255)
    54.198.0.0/16 (54.198.0.0 – 54.198.255.255)
    54.80.0.0/13 (54.80.0.0 – 54.87.255.255)
    54.88.0.0/14 (54.88.0.0 – 54.91.255.255)
    54.92.0.0/16 (54.92.0.0 – 54.92.255.255)
    54.92.128.0/17 (54.92.128.0 – 54.92.255.255)

    US West (Oregon):

    50.112.0.0/16 (50.112.0.0 – 50.112.255.255)
    54.245.0.0/16 (54.245.0.0 – 54.245.255.255)
    54.244.0.0/16 (54.244.0.0 – 54.244.255.255)
    54.214.0.0/16 (54.214.0.0 – 54.214.255.255)
    54.212.0.0/15 (54.212.0.0 – 54.213.255.255)
    54.218.0.0/16 (54.218.0.0 – 54.218.255.255)
    54.200.0.0/15 (54.200.0.0 – 54.201.255.255)
    54.202.0.0/15 (54.202.0.0 – 54.203.255.255)
    54.184.0.0/13 (54.184.0.0 – 54.191.255.255)

    US West (Northern California):

    204.236.128.0/18 (204.236.128.0 – 204.236.191.255)
    184.72.0.0/18 (184.72.0.0 – 184.72.63.255)
    50.18.0.0/16 (50.18.0.0 – 50.18.255.255)
    184.169.128.0/17 (184.160.128.0 – 184.169.255.255)
    54.241.0.0/16 (54.241.0.0 – 54.241.255.255)
    54.215.0.0/16 (54.215.0.0 – 54.215.255.255)
    54.219.0.0/16 (54.219.0.0 – 54.219.255.255)
    54.193.0.0/16 (54.193.0.0 – 54.193.255.255)
    54.176.0.0/15 (54.176.0.0 – 54.177.255.255)
    54.183.0.0/16 (54.183.0.0 – 54.183.255.255)

    EU (Ireland):

    79.125.0.0/17 (79.125.0.0 – 79.125.127.255)
    46.51.128.0/18 (46.51.128.0 – 46.51.191.255)
    46.51.192.0/20 (46.51.192.0 – 46.51.207.255)
    46.137.0.0/17 (46.137.0.0 – 46.137.127.255)
    46.137.128.0/18 (46.137.128.0 – 46.137.191.255)
    176.34.128.0/17 (176.34.128.0 – 176.34.255.255)
    176.34.64.0/18 (176.34.64.0 – 176.34.127.255)
    54.247.0.0/16 (54.247.0.0 – 54.247.255.255)
    54.246.0.0/16 (54.246.0.0 – 54.246.255.255)
    54.228.0.0/16 (54.228.0.0 – 54.228.255.255)
    54.216.0.0/15 (54.216.0.0 – 54.217.255.255)
    54.229.0.0/16 (54.229.0.0 – 54.229.255.255)
    54.220.0.0/16 (54.220.0.0 – 54.220.255.255)
    54.194.0.0/15 (54.194.0.0 – 54.195.255.255)
    54.72.0.0/14 (54.72.0.0 – 54.75.255.255)
    54.76.0.0/15 (54.76.0.0 – 54.77.255.255)
    54.78.0.0/16 (54.78.0.0 – 54.78.255.255)
    54.74.0.0/15 (54.74.0.0 – 54.75.255.255)
    185.48.120.0/22 (185.48.120.0 – 185.48.123.255)

    Asia Pacific (Singapore)

    175.41.128.0/18 (175.41.128.0 – 175.41.191.255)
    122.248.192.0/18 (122.248.192.0 – 122.248.255.255)
    46.137.192.0/18 (46.137.192.0 – 46.137.255.255)
    46.51.216.0/21 (46.51.216.0 – 46.51.223.255)
    54.251.0.0/16 (54.251.0.0 – 54.251.255.255)
    54.254.0.0/16 (54.254.0.0 – 54.254.255.255)
    54.255.0.0/16 (54.255.0.0 – 54.255.255.255)
    54.179.0.0/16 (54.179.0.0 – 54.179.255.255)

    Asia Pacific (Sydney)

    54.252.0.0/16 (54.252.0.0 – 54.252.255.255)
    54.253.0.0/16 (54.253.0.0 – 54.253.255.255)
    54.206.0.0/16 (54.206.0.0 – 54.206.255.255)
    54.79.0.0/16 (54.79.0.0 – 54.79.255.255)

    Asia Pacific (Tokyo)

    175.41.192.0/18 (175.41.192.0 – 175.41.255.255)
    46.51.224.0/19 (46.51.224.0 – 46.51.255.255)
    176.32.64.0/19 (176.32.64.0 – 176.32.95.255)
    103.4.8.0/21 (103.4.8.0 – 103.4.15.255)
    176.34.0.0/18 (176.34.0.0 – 176.34.63.255)
    54.248.0.0/15 (54.248.0.0 – 54.249.255.255)
    54.250.0.0/16 (54.250.0.0 – 54.250.255.255)
    54.238.0.0/16 (54.238.0.0 – 54.238.255.255)
    54.199.0.0/16 (54.199.0.0 – 54.199.255.255)
    54.178.0.0/16 (54.178.0.0 – 54.178.255.255)
    54.95.0.0/16 (54.95.0.0-54.95.255.255)
    54.92.0.0/17 (54.92.0.0 – 54.92.127.255)

     
    We obtained these ranges from Amazon’s EC2 list of public IP ranges.

  • Other than IP address, is there any way to identify LoadStorm requests from normal traffic?

    Yes, there are two additional ways for you to white list our traffic:

    1. The user agent in the HTTP header includes LoadStorm. However, if your test scripts contain steps with HTTPS requests, then those headers will be encrypted.
    2. If you are not testing against your production server, we recommend trying a non-standard port number to identify the traffic as load testing. LoadStorm allows you to use any port number and can be designated for each target server.

    Thumbs Up Button

  • When configuring a scenario step that clicks a link, how does LoadStorm follow the link?

    LoadStorm follows links when the session ID is appended to the URL query string because it searches for the text between the open and close anchor tags. If it doesn’t find that text, then the fall back is to match the XPATH. We stored the XPATH when the scenario was created (essentially an address inside the page’s DOM). The XPATH is a unique identifier that allows us to click on the anchor such as an image or the href attribute or execute the Javascript.

     

  • What is the “Average Duration” for a scenario?

    Average Duration is the time taken for ALL steps to complete (see step duration calculation), plus the average pause times between steps. Thus, it is the mean time for a specific vuser’s “life”. The actual duration fluctuates with the amount of pause assigned to that vuser. The Average Duration is shown above the steps on the Scenario page.

     

  • How is “Duration” for a scenario step calculated?

    Duration is the time taken for a scenario step to complete, which is the sum of time to last byte for each request in the step (including HTML, JS, CSS, images, etc.). The Average Duration shown above the steps on the Scenario page is the total of all step durations plus the average pause times between steps.

     

  • What does “response time” mean?

    load testing infoLoadStorm calculates the response time in the following manner:

    The total time taken between the first byte of the request sent and the last byte of the response received for all requests made. This includes the network latency and the time for the server to respond.

    The primary page normally includes the HTML, CSS, and Javascript files. However, the primary page could be another resource such as a PDF, video, or Flash file. The response time may appear slightly faster than observed by you in a browser because it does not include time for a browser to render the page. Of course render time will be affected by browser type, operating system, and horsepower of the client computer.

    It is also important to note that LoadStorm’s servers probably have faster internet connections than many homes and offices.

     

  • Can I create a test script where virtual users login to my web application? If so, how do I get my test user file into LoadStorm?

    Yes. Once you’ve created a script, you may modify forms in order to log into your application with test user data.

    We recommend you visit our learning center, where we walk you through the parameterization process for a script. A brief explanation is given below.

    While creating a recording using developer tools in your browser, log in to your application. Save the recording as a HAR file and upload it into your account. Open your script and select the parameterization tab.

    After that, upload your CSV which contains necessary login info (username/email, passwords) into the Build… User Data section of LoadStorm. Note that the first row’s data will be read as column headers and not row data. You can also choose to “generate data” on-the-fly in LoadStorm. If you need to embed a comma inside a particular data field, simply place quotation marks around all field data in the file.

    Finally, edit the necessary script and find the request that submits the form. Modify the form to include test user information. To add the test user data, you must select “custom” under the form modifications options, and select the data from a CSV file.

    This simulates traffic more realistically. Instead of simulating one user logging in over and over, many different users log in instead.

     

  • How does it generate HTTP traffic against my web application?

    LoadStorm is a truly distributed application that leverages the power of Amazon Web Services to scale on demand with processing power and bandwidth as needed to test the largest web projects. As you crank up the testing load from 200 to 150,000 virtual users, LoadStorm is automatically adding machines for you (as many as necessary) from Amazon’s server farm to handle the processing. When your tests are done and the extra machines are not needed, they are turned off to wait for another test.

    The normal processing is for a script to be processed as a sequential series of pages that you have defined in the recording. Once a page finishes, the VUser pauses for the specified time (between Min & Max “Think Time” in script), then the next page proceeds. After the last page completes, the VUser context (session, cache, cookies) is cleared and the script starts from the beginning again with a new VUser.

    You define the number of concurrent users you want to start with and peak at. As one VUser completes the script as described above, then another VUser is created to replace it – starting at page 1 again.

    If you have started at a small number and set the peak number much higher, then LoadStorm will not only retire VUsers that have completed the script, it will also add additional users throughout the test to ramp up the volume.
    This is repeated until the test is done as defined by the test duration field (in minutes) when you began the load test.

     

  • What is the “Form data set” option when creating a new scenario? Why are there options for Tiny, Small, and Huge?

    The Form Data Set field that you see on the Scenario settings page is a mechanism to select an input source for form submission. Using the Form Data Sets functionality, it is possible to submit forms such as a login form and fill-in the fields (e.g. username, password) with data from your system. Other common types of form submission are search terms, credit card numbers, and profile preferences. Any type of form field that can be filled in with text can utilize the Form Data Sets.

    You may upload your own data to be used for form submission, or you may use one of our built-in form data sets.

    The Tiny, Small, and Huge form data sets are built-into LoadStorm. The only real difference between the three files are the number of rows. Tiny only has 10 records. Small has 100, and Huge has 10,000 records of user information. They contain user information:

    • First Name
    • Last Name
    • Username
    • Password
    • Email address

    Many of our customers use one of our built-in form data sets for registration on their web application or for logging in virtual users.

    If you are going to upload your own data into LoadStorm, here is an example of the process for getting user credentials from your system to LoadStorm:

    1. Export a CSV file of user information from your user table(s).
    2. The first row of the CSV must contain the names of the columns, and these column names will be mapped to the field name in the step that submits the form.
    3. The CSV file is uploaded by clicking on the UPLOAD DATA button on the Build tab. This button will be near the bottom, under the Form Data Sets section of the page.
    4. You must give the file a label so that you can choose that file to be used at the scenario level.

    Once you have the CSV data in LoadStorm (you should see it on the Build tab under the heading Form Data Sets), then you need to edit the scenario and choose the appropriate data set based on the label you assigned when uploading it. This essentially connects the CSV file to your scenario. Then when you add a step for form submission, you will be able to map the fields in the form to the columns of data in the Form Data Set.

    The LoadStorm Tour has a video tutorial showing how to create a login form submission. Watch the video on YouTube

     

  • How Can I Increase the Requests per Second in My Test?

    Here are some thoughts on increasing RPS in LoadStorm tests:

    1. Reducing think time between pages in a script – minimum is 5 second pause, so set both min & max to 5.
    2. Increase the number of concurrent users.
    3. Reduce the response time from your server which decreases LoadStorm’s waiting for a response.

    RPS will always be impacted by the types of pages/resources your scripts are requesting. The way it works is that LoadStorm cranks up the starting concurrent users shortly after test starts. It then walks each user through the script assigned to it. One step is executed (such as home page), and all the images, JS, CSS, XML are requested separately. When those responses are all returned, then the system pauses a random time between the Min & Max “Think Time”, then it requests the next page. That continues until the end of the script, whereupon that virtual user is retired. After a few seconds, another vuser is created and the process starts again. So, having more users going at the same time, without waiting more than a few hundred ms to get a response, will increase RPS.

     

  • Can I change the host file for my test? I need to hit a server that does NOT currently have a valid DNS name because we don’t want it found by customers until we launch the site.

    Unfortunately, our load testing tool doesn’t have a feature for what you need. Normally this is a simple process of putting a domain name to IP address mapping in a configuration file on a particular machine. The process is not as easy in our situation because we use dynamically instantiated servers from the Amazon cloud; thus, we do not know what IP addresses we will be using before you run your tests. We would need to create new images specifically for your tests, and we would need to make changes to our core launching software that instantiates the images so that new machines are launched with this particular host mapping. If you would like us to coordinate this custom process, please contact our support. Additional charges per test will apply.

     

  • If I stop a test before my max users is reached, are the rest of my users wasted?

    Occasionally, a test will be terminated early because the target server fails or the tester stops it. It is not uncommon to have a test automatically stopped by LoadStorm when error rates hit 70% or so. When this happens in a test using virtual users, those users may seem “wasted”.

    Customers have asked us why the system deducted 5,000 VUsers from their account when the test only lasted 10 minutes and stopped at 500 users. The reason LoadStorm does not prorate the VUsers to the minute or to the actual peak users is because we incur the full costs of the 5,000 VUser test.

    Amazon’s cloud is elastic, and we “rent” the servers needed for your tests. However, there is a minimum of 1 hour increments that Amazon will bill us. For example, if your test needs 20 servers to hit your defined peak users, we buy those 20 servers upfront before your test actually starts. Then if you load test dies 5 minutes into it, we are still charged as if the test used all 20 servers for 1 hour.

    Thus, we recommend that you start with smaller tests to verify that you have all of the environmental factors properly configured. Growing the volume in several tests that increase in volume will potentially eliminate the “wasted” feeling of throwing 5,000 users at a server that fails at 500 users.

     

  • Why not let our users test the system’s capacity?

    For some projects, this is an acceptable approach. For some internal applications, apps that are very simple will realistically expect light traffic. The best plan may be to get it done as quickly as possible. You have to ask yourself what is the risk of the application not working at some point. For many businesses, getting a large influx of new users is the peak of success, and that success would be shattered by having the new users unable to use the system, unable to buy your product and unhappy with your company.

     

  • How do I stop a test that has killed my server?

    load testing stopYou can press the “STOP NOW” button at any time. It is found on the Run tab, and the button will only appear during the execution of a load test.

    Also, LoadStorm will automatically stop a load test if it can determine that your server has been overloaded. The thresholds for stopping a test are currently when the Error Rate exceeds 50% or the Average Response Time exceeds 15 seconds for the most recently summarized test period.

    Important Note for Small Tests: The sample interval is not the entire test but just the most recent part of it, and there must be a minimum number of requests in the sample. This interval is significant because in very small tests the number of failed requests AND the total requests may be few. For example, you must have some requests for at least 2 intervals, which means you have summarized more than 1 minute of requests.

     

  • Why does the Summary Table of test results show my missing JPGs in the HTML row?

    If one of your web pages contains a reference to an image that is not available on your system, then a response header will be sent to LoadStorm have Status Code = 404. Additionally, your web server will send an HTML page indicating the 404 “Not Found” page to alert the user. That’s standard practice in web applications.

    The reason LoadStorm reports the 404 errors as HTML is because your web server is sending our system the Content Type = “HTML” attribute. LoadStorm separates the data in the HTML and Other rows by the Content Type.

     

  • Why aren’t the 500 users evenly distributed across all 5 of my servers behind a round robin load balancer?

    A round robin load balancer will not work very well for testing your configuration unless you are using a much higher number of virtual users (at least 2000 or so). Each of our simulation servers will resolve the DNS name independently of the others, but each of our servers will simulate hundreds of users. I would expect a peak load of 500 users to get routed to more than one of your servers, but probably not all 5 of them. A peak load of 100 users or less is likely to hit only one of your servers.

     

  • Will the load test show up in the Google Analytics reports? How is testing typically tracked by Google Analytics?

    We would not expect LoadStorm test volume to show up in your Google Analytics. We do execute the javascript, but requests outside of your domain are blocked unless those other servers have also been verified. Servers like Google Analytics should be set to “Ignore”. So, the tracking by Analytics, ad servers, or other systems outside of your domain would not work. Some of these kinds of services have restrictions against load testing.

    Almost all of our customers don’t want our load testing tool traffic to show up in their marketing statistics (Google Analytics) because it essentially renders their analytics data worthless. Additionally, Google has limits on the amount of hits per period in their free GA account; otherwise you must upgrade to the paying account. There are also legal issues involved whereby Google doesn’t want automated tools like LoadStorm hammering their applications (even GA) – it is a violation of their standard terms and conditions. Thus, when a GA tracking javascript is found in a page that is involved in a LoadStorm test, that server is ignored automatically by our tool.

    Most of our clients use a different mechanism for watching the user activity of a load test. There are many tools for monitoring the website, including several excellent options for open source and commercial products such as New Relic, Loggly, and Hyperic. Most of these are very inexpensive or free, up to a certain level. These types of tools provide significant information regarding the user activity as well as important data on your architecture. For example, is your web server memory bound when response times start to increase drastically? Or did your database server hit 99% utilization when throughput drops?

    New Relic and Hyperic can monitor server activity from the local server using an agent. Loggly is a tool that has teamed up with New Relic for analyzing server logs which can make your life easier when trying to extrapolate the user activity from your server log.

    So to sum it up any javascript that reports traffic to third-party analytic tools will not work with LoadStorm. Instead useful tools for monitoring server usage and analyzing log files of user activity on the server under test (SUT) our encouraged and will work with LoadStorm.

  • Can your system run load tests behind my firewall for non-Internet applications?

    Unfortunately not. LoadStorm is only for web applications that can be reached from the Amazon cloud. If you have an intranet application, then opening a hole in your firewall will allow LoadStorm to run load tests against it.

     

  • How much does LoadStorm emulate the caching abilities of a real browser client? Will cache be reset across various sessions?

    The short answer is: If your browser caches it, LoadStorm will cache it for a virtual user. When a virtual user completes a script, it is retired which causes the session to end.

    Our load testing tool is designed to closely emulate the browser with a new, empty cache between sessions. So, generally your static assets will be cached for later pages in a script according to their header values. There are a few exceptions. We do not cache HTML resources since these are usually dynamically generated. Most LoadStorm users expect a subsequent HTML request to get this file again, thus that is how we coded it to work. If the headers are set properly, this will usually happen anyhow.

     

  • Does LoadStorm download images as a part of the load testing?

    Yes, and only if you want to. LoadStorm has a checkbox on the scenario edit page to turn download images on or off. There is a similar option for controlling the download and execution of Javascript files.

    Normally performance issues arise in the process of generating the dynamic page, and web servers can serve static images very quickly; however, we understand that many of our clients have websites with a large amount of media-related content, and load test results from those sites could be significantly affected by not downloading that content. Image download is especially relevant to performance if bottlenecks are in bandwidth or other network communications.

     

  • What’s the difference between “Concurrent Users”, “Sessions”, and “Total Users”?

    load testing browser emulationThe starting and peak number of virtual users in LoadStorm represent concurrent users or the number of simulated users at a particular point in time. Each concurrent user will last for the duration of the script. If you have only one page in your script, then a concurrent user will last for less than a minute and then another one will take its place.

    For example, if you have a test with one script having one page, starting at 10 concurrent users and peak of 10 concurrent users for 20 minutes, there will be 200-300 total users simulated (and same number of sessions). LoadStorm will create a new session for each virtual user, but it will not maintain the session after the script has ended, so your session should time out as you have configured it.

    The LoadStorm test parameters simulate “concurrent” users not total users. The total simulated users will generally be much higher than concurrent users, but the actual count depends on lots of other factors such as the number of pages in the script, think times, etc.

     

  • Why am I not getting as many page views as expected for the number of concurrent users in my test?

    Page views is a good term when reviewing Google Analytics data for traffic analysis. However, when reviewing load test data, page views can be a confusing metric. We do not track page views; rather we track requests and responses. Let’s look at the difference and how LoadStorm measures interactions with the server.

    The page views per virtual user should stay consistent as long as the response time is consistent and no errors. In some tests, the response time and errors are significant and increase over the duration of the test. For example, we see some load tests that have a large number of timeouts (no response to the request in 35 seconds). That means that LoadStorm cannot move to the next request for a virtual user until the response is received or we timeout the request. This drastically changes the equation you may be using to calculate the number of pageviews. Specifically, in one test there were 143,553 errors out of 1,345,534 total requests during a 30 minute test. This failure rate greatly decreased the number of requests LoadStorm was sending during the test.

    Also, if a request is sent for a step, LoadStorm is using the response header to capture the content type. A normal HTML page in a test returns, “text/html; charset=UTF-8″. When no response is received, LoadStorm assigns the response content type as, “NONE”. This affects the summary report table because the NONE responses are lumped into the OTHER category rather than the HTML category. Therefore, many of the 143k errors are shown in the OTHER category – not in the HTML category. That means that many of the HTML requests are not even shown in the table as HTML (or page views).

    The bottom line is that it’s mathematically invalid to expect linear correlation between page views, requests/responses, and vusers if the target server is not responding well. Everyone has a different definition of test success and failure. Some believe that greater than 1% error rate is a reason to stop the test. Others believe that an average response time greater than 3 seconds is a failure. Regardless how you define failure, timeouts and other server errors degrade the value of your load test metrics.

     

  • While I am running a load test and X number of users are hitting my web app, does this mean every second X number of requests are being sent to my application?

    The short answer is no. LoadStorm reporting graphs include a metric called Requests Per Second, and that is what you are describing.

    A virtual or simulated user does what a “real” user does as specified by the scripts and pages that you have created in LoadStorm. If there are X number of users, then there are X script instances running at that particular time.

    When creating a script, you can adjust the “Think Times” which is how much time there is between pages. The actual pause is randomly generated to give a more realistic load against the server. As an example, a script which opens a single page for each step with an average think time of 30 seconds will have an average of 2 requests per minute per user. It gets a bit more complicated when there are external files like Javascript. These are added as well and counted in the total requests. When one script ends for a designated number of users, another is started within a minute to maintain the desired user load until the end of the test.

     

  • What happens to a virtual user when something fails or times out in a scenario?

    load testing vuserThe vuser continues to move to the next step in its scenario. For instance, if a JavaScript file fails to download during step 3:

    • That error will be logged in LoadStorm for reporting
    • That request will be terminated
    • All other requests for step 3 will be attempted as usual
    • The vuser will begin step 4 as defined in the scenario

    If any particular resource times out within a step, some of the step processing may not finish. This depends on what kind of resource fails and at what point in the step. If an image fails, this is relatively harmless. The images following will not be requested, but everything else will continue normally. If the html fails with a timeout, then the step will stop and no images or other resources will be requested. Images, javascript and css all depend on the html, so none will be requested for that step unless the html is received without error.

    The step following a step with an error will work normally to the extent that it can. As with a browser, you cannot click on a link or submit a form if you do not currently have a valid page available with which to work. You can always type in a new url and go from there. Likewise, in LoadStorm if the next step is an open step, then previous steps will have no effect except for possibly login or cookie state. If the next step is a click or form, then the previous html must have been received for the step to function. If the step cannot work because the previous html timed out or otherwise failed, then that step will show no requests or errors and the next step will be attempted after the expected pause.

    There is currently only one situation where a scenario will restart before all of its steps have run. This is a step timeout where a step starts but does not finish before 5 minutes have passed. A step includes getting all resources within the step such as html, javascript, images, css, etc. So, it is not inconceivable that this timeout occurs if there are 50, 100 or more resources in a step, but it is long enough that it is pretty rare.

     

  • What happens to virtual users as test scripts are running?

    LoadStorm creates the starting concurrent users within seconds of the test starting. For example, if you set the test to start at 1,000 users then you will have requests coming from those 1,000 users in the first 30 seconds. It then processes each virtual user through the script assigned to it. The first page, for example your home page, is execute first. The HTML is sent from your server and LoadStorm receives it. All other resources recorded (images, Javascript, CSS, and XML) are requested separately and sequentially.

    When all responses in a page are returned, the system pauses a random time between Min & Max Think Time pause (you can edit this), then it requests the next page. That continues until the virtual user reaches the end of the script, whereupon that VUser is “retired”. After about 30 seconds, another VUser is created and the process starts again. The cache for the new virtual user starts empty and does not remember cookies or static resources for the previous VUser.

     

  • Does LoadStorm’s request header allow for compression such as gzip?

    There is currently no functionality to add gzip to the LoadStorm request header. We hope to add that feature in an upcoming release. “Accept-Encoding: gzip, deflate” cannot be sent in the request generated by LoadStorm.

     


Analyzing Test Results

  • What does “response time” mean?

    load testing infoLoadStorm calculates the response time in the following manner:

    The total time taken between the first byte of the request sent and the last byte of the response received for all requests made. This includes the network latency and the time for the server to respond.

    The primary page normally includes the HTML, CSS, and Javascript files. However, the primary page could be another resource such as a PDF, video, or Flash file. The response time may appear slightly faster than observed by you in a browser because it does not include time for a browser to render the page. Of course render time will be affected by browser type, operating system, and horsepower of the client computer.

    It is also important to note that LoadStorm’s servers probably have faster internet connections than many homes and offices.

     

  • How does it generate HTTP traffic against my web application?

    LoadStorm is a truly distributed application that leverages the power of Amazon Web Services to scale on demand with processing power and bandwidth as needed to test the largest web projects. As you crank up the testing load from 200 to 150,000 virtual users, LoadStorm is automatically adding machines for you (as many as necessary) from Amazon’s server farm to handle the processing. When your tests are done and the extra machines are not needed, they are turned off to wait for another test.

    The normal processing is for a script to be processed as a sequential series of pages that you have defined in the recording. Once a page finishes, the VUser pauses for the specified time (between Min & Max “Think Time” in script), then the next page proceeds. After the last page completes, the VUser context (session, cache, cookies) is cleared and the script starts from the beginning again with a new VUser.

    You define the number of concurrent users you want to start with and peak at. As one VUser completes the script as described above, then another VUser is created to replace it – starting at page 1 again.

    If you have started at a small number and set the peak number much higher, then LoadStorm will not only retire VUsers that have completed the script, it will also add additional users throughout the test to ramp up the volume.
    This is repeated until the test is done as defined by the test duration field (in minutes) when you began the load test.

     

  • Why aren’t the 500 users evenly distributed across all 5 of my servers behind a round robin load balancer?

    A round robin load balancer will not work very well for testing your configuration unless you are using a much higher number of virtual users (at least 2000 or so). Each of our simulation servers will resolve the DNS name independently of the others, but each of our servers will simulate hundreds of users. I would expect a peak load of 500 users to get routed to more than one of your servers, but probably not all 5 of them. A peak load of 100 users or less is likely to hit only one of your servers.

     

  • Will the load test show up in the Google Analytics reports? How is testing typically tracked by Google Analytics?

    We would not expect LoadStorm test volume to show up in your Google Analytics. We do execute the javascript, but requests outside of your domain are blocked unless those other servers have also been verified. Servers like Google Analytics should be set to “Ignore”. So, the tracking by Analytics, ad servers, or other systems outside of your domain would not work. Some of these kinds of services have restrictions against load testing.

    Almost all of our customers don’t want our load testing tool traffic to show up in their marketing statistics (Google Analytics) because it essentially renders their analytics data worthless. Additionally, Google has limits on the amount of hits per period in their free GA account; otherwise you must upgrade to the paying account. There are also legal issues involved whereby Google doesn’t want automated tools like LoadStorm hammering their applications (even GA) – it is a violation of their standard terms and conditions. Thus, when a GA tracking javascript is found in a page that is involved in a LoadStorm test, that server is ignored automatically by our tool.

    Most of our clients use a different mechanism for watching the user activity of a load test. There are many tools for monitoring the website, including several excellent options for open source and commercial products such as New Relic, Loggly, and Hyperic. Most of these are very inexpensive or free, up to a certain level. These types of tools provide significant information regarding the user activity as well as important data on your architecture. For example, is your web server memory bound when response times start to increase drastically? Or did your database server hit 99% utilization when throughput drops?

    New Relic and Hyperic can monitor server activity from the local server using an agent. Loggly is a tool that has teamed up with New Relic for analyzing server logs which can make your life easier when trying to extrapolate the user activity from your server log.

    So to sum it up any javascript that reports traffic to third-party analytic tools will not work with LoadStorm. Instead useful tools for monitoring server usage and analyzing log files of user activity on the server under test (SUT) our encouraged and will work with LoadStorm.

  • How much does LoadStorm emulate the caching abilities of a real browser client? Will cache be reset across various sessions?

    The short answer is: If your browser caches it, LoadStorm will cache it for a virtual user. When a virtual user completes a script, it is retired which causes the session to end.

    Our load testing tool is designed to closely emulate the browser with a new, empty cache between sessions. So, generally your static assets will be cached for later pages in a script according to their header values. There are a few exceptions. We do not cache HTML resources since these are usually dynamically generated. Most LoadStorm users expect a subsequent HTML request to get this file again, thus that is how we coded it to work. If the headers are set properly, this will usually happen anyhow.

     

  • What’s the difference between “Concurrent Users”, “Sessions”, and “Total Users”?

    load testing browser emulationThe starting and peak number of virtual users in LoadStorm represent concurrent users or the number of simulated users at a particular point in time. Each concurrent user will last for the duration of the script. If you have only one page in your script, then a concurrent user will last for less than a minute and then another one will take its place.

    For example, if you have a test with one script having one page, starting at 10 concurrent users and peak of 10 concurrent users for 20 minutes, there will be 200-300 total users simulated (and same number of sessions). LoadStorm will create a new session for each virtual user, but it will not maintain the session after the script has ended, so your session should time out as you have configured it.

    The LoadStorm test parameters simulate “concurrent” users not total users. The total simulated users will generally be much higher than concurrent users, but the actual count depends on lots of other factors such as the number of pages in the script, think times, etc.

     

  • While I am running a load test and X number of users are hitting my web app, does this mean every second X number of requests are being sent to my application?

    The short answer is no. LoadStorm reporting graphs include a metric called Requests Per Second, and that is what you are describing.

    A virtual or simulated user does what a “real” user does as specified by the scripts and pages that you have created in LoadStorm. If there are X number of users, then there are X script instances running at that particular time.

    When creating a script, you can adjust the “Think Times” which is how much time there is between pages. The actual pause is randomly generated to give a more realistic load against the server. As an example, a script which opens a single page for each step with an average think time of 30 seconds will have an average of 2 requests per minute per user. It gets a bit more complicated when there are external files like Javascript. These are added as well and counted in the total requests. When one script ends for a designated number of users, another is started within a minute to maintain the desired user load until the end of the test.

     

  • What happens to a virtual user when something fails or times out in a scenario?

    load testing vuserThe vuser continues to move to the next step in its scenario. For instance, if a JavaScript file fails to download during step 3:

    • That error will be logged in LoadStorm for reporting
    • That request will be terminated
    • All other requests for step 3 will be attempted as usual
    • The vuser will begin step 4 as defined in the scenario

    If any particular resource times out within a step, some of the step processing may not finish. This depends on what kind of resource fails and at what point in the step. If an image fails, this is relatively harmless. The images following will not be requested, but everything else will continue normally. If the html fails with a timeout, then the step will stop and no images or other resources will be requested. Images, javascript and css all depend on the html, so none will be requested for that step unless the html is received without error.

    The step following a step with an error will work normally to the extent that it can. As with a browser, you cannot click on a link or submit a form if you do not currently have a valid page available with which to work. You can always type in a new url and go from there. Likewise, in LoadStorm if the next step is an open step, then previous steps will have no effect except for possibly login or cookie state. If the next step is a click or form, then the previous html must have been received for the step to function. If the step cannot work because the previous html timed out or otherwise failed, then that step will show no requests or errors and the next step will be attempted after the expected pause.

    There is currently only one situation where a scenario will restart before all of its steps have run. This is a step timeout where a step starts but does not finish before 5 minutes have passed. A step includes getting all resources within the step such as html, javascript, images, css, etc. So, it is not inconceivable that this timeout occurs if there are 50, 100 or more resources in a step, but it is long enough that it is pretty rare.

     

  • What happens to virtual users as test scripts are running?

    LoadStorm creates the starting concurrent users within seconds of the test starting. For example, if you set the test to start at 1,000 users then you will have requests coming from those 1,000 users in the first 30 seconds. It then processes each virtual user through the script assigned to it. The first page, for example your home page, is execute first. The HTML is sent from your server and LoadStorm receives it. All other resources recorded (images, Javascript, CSS, and XML) are requested separately and sequentially.

    When all responses in a page are returned, the system pauses a random time between Min & Max Think Time pause (you can edit this), then it requests the next page. That continues until the virtual user reaches the end of the script, whereupon that VUser is “retired”. After about 30 seconds, another VUser is created and the process starts again. The cache for the new virtual user starts empty and does not remember cookies or static resources for the previous VUser.

     


Building Test Scenarios

  • Why do I need to verify my server? How do I do that?

    This is a very common question. In fact, a good example is the email this week from a new user stating, “Just tried to test your service and it came back with these comments:

    There were problems with the following fields:

    I am not sure what to do now…Why do I have to verify these servers?”

    Are You Load Testing Facebook?

    However, the error messages above indicate that his test plan will be hitting servers that aren’t verified (authorized for testing). Put another way, his scenarios contain requests that go to servers that aren’t his, and LoadStorm gets in trouble when our customers load test other people’s servers without their permission.

    Does he really want to hit Twitter and Facebook for his load test? If so, is he testing their ability to handle his load? If so, does he have their permission to hammer on their servers? Probably not. They tend to get upset when someone utilizes a cloud load testing tool to simulate thousands of VUsers against their social media platform. Please read the terms and conditions of use for those services.

    And where do those VUsers originate? LoadStorm. So who gets the phone call from their lawyers? We do.

    Prevent Denial of Service Attacks

    Our verification process is designed to prevent LoadStorm from being used as a Denial of Service (DOS) weapon. By placing a code on your home page that LoadStorm can request and confirm, our system confirms that you have the control over that server; thus, it is safe for our tool to hammer it with VUsers.

     

  • How should I set up scripts to simulate real traffic?

    Most people create several scripts to accurately reflect different types of users. Scripts execute simultaneously in order to hit your site with the correct mix of traffic. You control the weighting of each script by giving it a number relative to the others. For example, if you have 2 scripts and you give one a weighting of 4 and the other a weighting of 1, then the first will generate 80% of the traffic and the second 20%. This 80/20 could be realistic for a retail e-commerce site where most of the people (80%) are browsing the product catalog, while a minority of people (20%) are going through a buying experience. You can of course throw in a low-weight scenario to represent some internal employees adding products or issuing refunds.

    Another way of entering the weight of the traffic is to just enter the exact percentage you would like each script to carry. For the example about you would simply enter 80 and 20. The only thing you have to be very careful about is that the percentages add up to 100. Otherwise, it will work like the above description, which is different.

    There is no limit to the number of scripts you can create in a test plan. We encourage you to build as many as necessary to accurately reflect the real traffic patterns on your site. That will give you the most meaningful metrics for tuning your applications.

    We also simulate the user “think time” with random pauses between pages. Real users click on something, then read the page or fill in a form, and then click on something else. Thus, you don’t want to have a simulated virtual user clicking on something every .5 seconds – it would not give you meaningful results if you want to see how your site holds up in the real world.

    Also see: Virtual User Calculations about calculating the amount of virtual users you need for a realistic test.

     

  • Why should I ramp up my test load volume over time?

    It helps you with performance engineering. We recommend that you do not begin a large test at the maximum number of virtual users. Rather than run a test for 10 minutes beginning with 5,000 and ending with 5,000, we suggest that you perhaps start with 500 users and increase to 5,000 over an hour. This approach has the advantage of allowing your system to get all the parts working properly at smaller load (e.g. caches, threads, database connections) before the heavier volume starts exposing potential bottlenecks in your application.

    From what we’ve seen in thousands of load tests with LoadStorm, it is common to see useful patterns in the metrics before the peak traffic is reached, and these metrics point to areas of performance limitations that can be tuned.

    Load testing is invariably coupled with performance tuning – an iterative process of test/tune/test/tune. Therefore, we recommend you ramp the volume in order to get more value from the test results.

     

  • Other than IP address, is there any way to identify LoadStorm requests from normal traffic?

    Yes, there are two additional ways for you to white list our traffic:

    1. The user agent in the HTTP header includes LoadStorm. However, if your test scripts contain steps with HTTPS requests, then those headers will be encrypted.
    2. If you are not testing against your production server, we recommend trying a non-standard port number to identify the traffic as load testing. LoadStorm allows you to use any port number and can be designated for each target server.

    Thumbs Up Button

  • Can I create a test script where virtual users login to my web application? If so, how do I get my test user file into LoadStorm?

    Yes. Once you’ve created a script, you may modify forms in order to log into your application with test user data.

    We recommend you visit our learning center, where we walk you through the parameterization process for a script. A brief explanation is given below.

    While creating a recording using developer tools in your browser, log in to your application. Save the recording as a HAR file and upload it into your account. Open your script and select the parameterization tab.

    After that, upload your CSV which contains necessary login info (username/email, passwords) into the Build… User Data section of LoadStorm. Note that the first row’s data will be read as column headers and not row data. You can also choose to “generate data” on-the-fly in LoadStorm. If you need to embed a comma inside a particular data field, simply place quotation marks around all field data in the file.

    Finally, edit the necessary script and find the request that submits the form. Modify the form to include test user information. To add the test user data, you must select “custom” under the form modifications options, and select the data from a CSV file.

    This simulates traffic more realistically. Instead of simulating one user logging in over and over, many different users log in instead.

     

  • How Can I Increase the Requests per Second in My Test?

    Here are some thoughts on increasing RPS in LoadStorm tests:

    1. Reducing think time between pages in a script – minimum is 5 second pause, so set both min & max to 5.
    2. Increase the number of concurrent users.
    3. Reduce the response time from your server which decreases LoadStorm’s waiting for a response.

    RPS will always be impacted by the types of pages/resources your scripts are requesting. The way it works is that LoadStorm cranks up the starting concurrent users shortly after test starts. It then walks each user through the script assigned to it. One step is executed (such as home page), and all the images, JS, CSS, XML are requested separately. When those responses are all returned, then the system pauses a random time between the Min & Max “Think Time”, then it requests the next page. That continues until the end of the script, whereupon that virtual user is retired. After a few seconds, another vuser is created and the process starts again. So, having more users going at the same time, without waiting more than a few hundred ms to get a response, will increase RPS.

     

  • Can I change the host file for my test? I need to hit a server that does NOT currently have a valid DNS name because we don’t want it found by customers until we launch the site.

    Unfortunately, our load testing tool doesn’t have a feature for what you need. Normally this is a simple process of putting a domain name to IP address mapping in a configuration file on a particular machine. The process is not as easy in our situation because we use dynamically instantiated servers from the Amazon cloud; thus, we do not know what IP addresses we will be using before you run your tests. We would need to create new images specifically for your tests, and we would need to make changes to our core launching software that instantiates the images so that new machines are launched with this particular host mapping. If you would like us to coordinate this custom process, please contact our support. Additional charges per test will apply.

     

  • Does LoadStorm’s request header allow for compression such as gzip?

    There is currently no functionality to add gzip to the LoadStorm request header. We hope to add that feature in an upcoming release. “Accept-Encoding: gzip, deflate” cannot be sent in the request generated by LoadStorm.

     


Getting Started

  • What Does LoadStorm Do?

    load testing infoLoadStorm™ is a web-based load testing tool for simulating what users do with a web site or web application. You use it to build tests that send requests to your server in the same way that a user’s browser sends requests to your server. But these tests are executed by our automated systems rather than by a user, so they can be done repeatedly and in large numbers simultaneously. They can also be built using our tool in such a way as to simulate a large number of different users with different tasks to perform.

     

  • How much does it cost?

    load testing costIt depends on how many simulated users you need for testing, and it depends on if you would like to use our consulting services.

    Creating an account is free, but the only limit of a free account is 50 concurrent users.

    For more details, please see Load Testing Cost.

  • Why do I need to verify my server? How do I do that?

    This is a very common question. In fact, a good example is the email this week from a new user stating, “Just tried to test your service and it came back with these comments:

    There were problems with the following fields:

    I am not sure what to do now…Why do I have to verify these servers?”

    Are You Load Testing Facebook?

    However, the error messages above indicate that his test plan will be hitting servers that aren’t verified (authorized for testing). Put another way, his scenarios contain requests that go to servers that aren’t his, and LoadStorm gets in trouble when our customers load test other people’s servers without their permission.

    Does he really want to hit Twitter and Facebook for his load test? If so, is he testing their ability to handle his load? If so, does he have their permission to hammer on their servers? Probably not. They tend to get upset when someone utilizes a cloud load testing tool to simulate thousands of VUsers against their social media platform. Please read the terms and conditions of use for those services.

    And where do those VUsers originate? LoadStorm. So who gets the phone call from their lawyers? We do.

    Prevent Denial of Service Attacks

    Our verification process is designed to prevent LoadStorm from being used as a Denial of Service (DOS) weapon. By placing a code on your home page that LoadStorm can request and confirm, our system confirms that you have the control over that server; thus, it is safe for our tool to hammer it with VUsers.

     

  • How should I set up scripts to simulate real traffic?

    Most people create several scripts to accurately reflect different types of users. Scripts execute simultaneously in order to hit your site with the correct mix of traffic. You control the weighting of each script by giving it a number relative to the others. For example, if you have 2 scripts and you give one a weighting of 4 and the other a weighting of 1, then the first will generate 80% of the traffic and the second 20%. This 80/20 could be realistic for a retail e-commerce site where most of the people (80%) are browsing the product catalog, while a minority of people (20%) are going through a buying experience. You can of course throw in a low-weight scenario to represent some internal employees adding products or issuing refunds.

    Another way of entering the weight of the traffic is to just enter the exact percentage you would like each script to carry. For the example about you would simply enter 80 and 20. The only thing you have to be very careful about is that the percentages add up to 100. Otherwise, it will work like the above description, which is different.

    There is no limit to the number of scripts you can create in a test plan. We encourage you to build as many as necessary to accurately reflect the real traffic patterns on your site. That will give you the most meaningful metrics for tuning your applications.

    We also simulate the user “think time” with random pauses between pages. Real users click on something, then read the page or fill in a form, and then click on something else. Thus, you don’t want to have a simulated virtual user clicking on something every .5 seconds – it would not give you meaningful results if you want to see how your site holds up in the real world.

    Also see: Virtual User Calculations about calculating the amount of virtual users you need for a realistic test.

     

  • Why should I ramp up my test load volume over time?

    It helps you with performance engineering. We recommend that you do not begin a large test at the maximum number of virtual users. Rather than run a test for 10 minutes beginning with 5,000 and ending with 5,000, we suggest that you perhaps start with 500 users and increase to 5,000 over an hour. This approach has the advantage of allowing your system to get all the parts working properly at smaller load (e.g. caches, threads, database connections) before the heavier volume starts exposing potential bottlenecks in your application.

    From what we’ve seen in thousands of load tests with LoadStorm, it is common to see useful patterns in the metrics before the peak traffic is reached, and these metrics point to areas of performance limitations that can be tuned.

    Load testing is invariably coupled with performance tuning – an iterative process of test/tune/test/tune. Therefore, we recommend you ramp the volume in order to get more value from the test results.

     

  • Why doesn’t my test start immediately?

    LoadStorm needs a little time to prepare the test resources such as the load generation servers. Our system is using each of your scripts and steps to pre-calculate the virtual users’ actions, how many servers will need to be instantiated from the cloud, random think times, etc. This produces essentially a test roadmap that our Scheduler and Summarizer modules use to coordinate all the HTTP traffic and correctly capture the metrics. Depending on the number of concurrent users and the duration of your test, the preparation of a load test can take 2 to 15 minutes.

     

  • What are the IP address ranges used by LoadStorm? We need to white list them.

    Because we use the Amazon EC2 cloud to dynamically instantiate load generation servers, we don’t have a specific static IP address, but here are the ranges:

    US East (Northern Virginia):

    72.44.32.0/19 (72.44.32.0 – 72.44.63.255)
    67.202.0.0/18 (67.202.0.0 – 67.202.63.255)
    75.101.128.0/17 (75.101.128.0 – 75.101.255.255)
    174.129.0.0/16 (174.129.0.0 – 174.129.255.255)
    204.236.192.0/18 (204.236.192.0 – 204.236.255.255)
    184.73.0.0/16 (184.73.0.0 – 184.73.255.255)
    184.72.128.0/17 (184.72.128.0 – 184.72.255.255)
    184.72.64.0/18 (184.72.64.0 – 184.72.127.255)
    50.16.0.0/15 (50.16.0.0 – 50.17.255.255)
    50.19.0.0/16 (50.19.0.0 – 50.19.255.255)
    107.20.0.0/14 (107.20.0.0 – 107.23.255.255)
    23.20.0.0/14 (23.20.0.0 – 23.23.255.255)
    54.242.0.0/15 (54.242.0.0 – 54.243.255.255)
    54.234.0.0/15 (54.234.0.0 – 54.235.255.255)
    54.236.0.0/15 (54.236.0.0 – 54.237.255.255)
    54.224.0.0/15 (54.224.0.0 – 54.225.255.255)
    54.226.0.0/15 (54.226.0.0 – 54.227.255.255)
    54.208.0.0/15 (54.208.0.0 – 54.209.255.255)
    54.210.0.0/15 (54.210.0.0 – 54.211.255.255)
    54.221.0.0/16 (54.221.0.0 – 54.221.255.255)
    54.204.0.0/15 (54.204.0.0 – 54.205.255.255)
    54.196.0.0/15 (54.196.0.0 – 54.197.255.255)
    54.198.0.0/16 (54.198.0.0 – 54.198.255.255)
    54.80.0.0/13 (54.80.0.0 – 54.87.255.255)
    54.88.0.0/14 (54.88.0.0 – 54.91.255.255)
    54.92.0.0/16 (54.92.0.0 – 54.92.255.255)
    54.92.128.0/17 (54.92.128.0 – 54.92.255.255)

    US West (Oregon):

    50.112.0.0/16 (50.112.0.0 – 50.112.255.255)
    54.245.0.0/16 (54.245.0.0 – 54.245.255.255)
    54.244.0.0/16 (54.244.0.0 – 54.244.255.255)
    54.214.0.0/16 (54.214.0.0 – 54.214.255.255)
    54.212.0.0/15 (54.212.0.0 – 54.213.255.255)
    54.218.0.0/16 (54.218.0.0 – 54.218.255.255)
    54.200.0.0/15 (54.200.0.0 – 54.201.255.255)
    54.202.0.0/15 (54.202.0.0 – 54.203.255.255)
    54.184.0.0/13 (54.184.0.0 – 54.191.255.255)

    US West (Northern California):

    204.236.128.0/18 (204.236.128.0 – 204.236.191.255)
    184.72.0.0/18 (184.72.0.0 – 184.72.63.255)
    50.18.0.0/16 (50.18.0.0 – 50.18.255.255)
    184.169.128.0/17 (184.160.128.0 – 184.169.255.255)
    54.241.0.0/16 (54.241.0.0 – 54.241.255.255)
    54.215.0.0/16 (54.215.0.0 – 54.215.255.255)
    54.219.0.0/16 (54.219.0.0 – 54.219.255.255)
    54.193.0.0/16 (54.193.0.0 – 54.193.255.255)
    54.176.0.0/15 (54.176.0.0 – 54.177.255.255)
    54.183.0.0/16 (54.183.0.0 – 54.183.255.255)

    EU (Ireland):

    79.125.0.0/17 (79.125.0.0 – 79.125.127.255)
    46.51.128.0/18 (46.51.128.0 – 46.51.191.255)
    46.51.192.0/20 (46.51.192.0 – 46.51.207.255)
    46.137.0.0/17 (46.137.0.0 – 46.137.127.255)
    46.137.128.0/18 (46.137.128.0 – 46.137.191.255)
    176.34.128.0/17 (176.34.128.0 – 176.34.255.255)
    176.34.64.0/18 (176.34.64.0 – 176.34.127.255)
    54.247.0.0/16 (54.247.0.0 – 54.247.255.255)
    54.246.0.0/16 (54.246.0.0 – 54.246.255.255)
    54.228.0.0/16 (54.228.0.0 – 54.228.255.255)
    54.216.0.0/15 (54.216.0.0 – 54.217.255.255)
    54.229.0.0/16 (54.229.0.0 – 54.229.255.255)
    54.220.0.0/16 (54.220.0.0 – 54.220.255.255)
    54.194.0.0/15 (54.194.0.0 – 54.195.255.255)
    54.72.0.0/14 (54.72.0.0 – 54.75.255.255)
    54.76.0.0/15 (54.76.0.0 – 54.77.255.255)
    54.78.0.0/16 (54.78.0.0 – 54.78.255.255)
    54.74.0.0/15 (54.74.0.0 – 54.75.255.255)
    185.48.120.0/22 (185.48.120.0 – 185.48.123.255)

    Asia Pacific (Singapore)

    175.41.128.0/18 (175.41.128.0 – 175.41.191.255)
    122.248.192.0/18 (122.248.192.0 – 122.248.255.255)
    46.137.192.0/18 (46.137.192.0 – 46.137.255.255)
    46.51.216.0/21 (46.51.216.0 – 46.51.223.255)
    54.251.0.0/16 (54.251.0.0 – 54.251.255.255)
    54.254.0.0/16 (54.254.0.0 – 54.254.255.255)
    54.255.0.0/16 (54.255.0.0 – 54.255.255.255)
    54.179.0.0/16 (54.179.0.0 – 54.179.255.255)

    Asia Pacific (Sydney)

    54.252.0.0/16 (54.252.0.0 – 54.252.255.255)
    54.253.0.0/16 (54.253.0.0 – 54.253.255.255)
    54.206.0.0/16 (54.206.0.0 – 54.206.255.255)
    54.79.0.0/16 (54.79.0.0 – 54.79.255.255)

    Asia Pacific (Tokyo)

    175.41.192.0/18 (175.41.192.0 – 175.41.255.255)
    46.51.224.0/19 (46.51.224.0 – 46.51.255.255)
    176.32.64.0/19 (176.32.64.0 – 176.32.95.255)
    103.4.8.0/21 (103.4.8.0 – 103.4.15.255)
    176.34.0.0/18 (176.34.0.0 – 176.34.63.255)
    54.248.0.0/15 (54.248.0.0 – 54.249.255.255)
    54.250.0.0/16 (54.250.0.0 – 54.250.255.255)
    54.238.0.0/16 (54.238.0.0 – 54.238.255.255)
    54.199.0.0/16 (54.199.0.0 – 54.199.255.255)
    54.178.0.0/16 (54.178.0.0 – 54.178.255.255)
    54.95.0.0/16 (54.95.0.0-54.95.255.255)
    54.92.0.0/17 (54.92.0.0 – 54.92.127.255)

     
    We obtained these ranges from Amazon’s EC2 list of public IP ranges.

  • If I stop a test before my max users is reached, are the rest of my users wasted?

    Occasionally, a test will be terminated early because the target server fails or the tester stops it. It is not uncommon to have a test automatically stopped by LoadStorm when error rates hit 70% or so. When this happens in a test using virtual users, those users may seem “wasted”.

    Customers have asked us why the system deducted 5,000 VUsers from their account when the test only lasted 10 minutes and stopped at 500 users. The reason LoadStorm does not prorate the VUsers to the minute or to the actual peak users is because we incur the full costs of the 5,000 VUser test.

    Amazon’s cloud is elastic, and we “rent” the servers needed for your tests. However, there is a minimum of 1 hour increments that Amazon will bill us. For example, if your test needs 20 servers to hit your defined peak users, we buy those 20 servers upfront before your test actually starts. Then if you load test dies 5 minutes into it, we are still charged as if the test used all 20 servers for 1 hour.

    Thus, we recommend that you start with smaller tests to verify that you have all of the environmental factors properly configured. Growing the volume in several tests that increase in volume will potentially eliminate the “wasted” feeling of throwing 5,000 users at a server that fails at 500 users.

     

  • Why not let our users test the system’s capacity?

    For some projects, this is an acceptable approach. For some internal applications, apps that are very simple will realistically expect light traffic. The best plan may be to get it done as quickly as possible. You have to ask yourself what is the risk of the application not working at some point. For many businesses, getting a large influx of new users is the peak of success, and that success would be shattered by having the new users unable to use the system, unable to buy your product and unhappy with your company.

     


Only LoadStorm 1.5

  • When configuring a scenario step that clicks a link, how does LoadStorm follow the link?

    LoadStorm follows links when the session ID is appended to the URL query string because it searches for the text between the open and close anchor tags. If it doesn’t find that text, then the fall back is to match the XPATH. We stored the XPATH when the scenario was created (essentially an address inside the page’s DOM). The XPATH is a unique identifier that allows us to click on the anchor such as an image or the href attribute or execute the Javascript.

     

  • What is the “Average Duration” for a scenario?

    Average Duration is the time taken for ALL steps to complete (see step duration calculation), plus the average pause times between steps. Thus, it is the mean time for a specific vuser’s “life”. The actual duration fluctuates with the amount of pause assigned to that vuser. The Average Duration is shown above the steps on the Scenario page.

     

  • How is “Duration” for a scenario step calculated?

    Duration is the time taken for a scenario step to complete, which is the sum of time to last byte for each request in the step (including HTML, JS, CSS, images, etc.). The Average Duration shown above the steps on the Scenario page is the total of all step durations plus the average pause times between steps.

     

  • What is the “Form data set” option when creating a new scenario? Why are there options for Tiny, Small, and Huge?

    The Form Data Set field that you see on the Scenario settings page is a mechanism to select an input source for form submission. Using the Form Data Sets functionality, it is possible to submit forms such as a login form and fill-in the fields (e.g. username, password) with data from your system. Other common types of form submission are search terms, credit card numbers, and profile preferences. Any type of form field that can be filled in with text can utilize the Form Data Sets.

    You may upload your own data to be used for form submission, or you may use one of our built-in form data sets.

    The Tiny, Small, and Huge form data sets are built-into LoadStorm. The only real difference between the three files are the number of rows. Tiny only has 10 records. Small has 100, and Huge has 10,000 records of user information. They contain user information:

    • First Name
    • Last Name
    • Username
    • Password
    • Email address

    Many of our customers use one of our built-in form data sets for registration on their web application or for logging in virtual users.

    If you are going to upload your own data into LoadStorm, here is an example of the process for getting user credentials from your system to LoadStorm:

    1. Export a CSV file of user information from your user table(s).
    2. The first row of the CSV must contain the names of the columns, and these column names will be mapped to the field name in the step that submits the form.
    3. The CSV file is uploaded by clicking on the UPLOAD DATA button on the Build tab. This button will be near the bottom, under the Form Data Sets section of the page.
    4. You must give the file a label so that you can choose that file to be used at the scenario level.

    Once you have the CSV data in LoadStorm (you should see it on the Build tab under the heading Form Data Sets), then you need to edit the scenario and choose the appropriate data set based on the label you assigned when uploading it. This essentially connects the CSV file to your scenario. Then when you add a step for form submission, you will be able to map the fields in the form to the columns of data in the Form Data Set.

    The LoadStorm Tour has a video tutorial showing how to create a login form submission. Watch the video on YouTube

     

  • Why does my step not show my images or contents aren’t displaying properly?

    When adding a step in LoadStorm, our emulator makes the request to your web application and receives the response containing the HTML code for the page. Our tool then parses the HTML to determine what additional resources (e.g. images, Javascript libraries) need to be requested for that page.

    Each of those resources are requested and received. LoadStorm saves all of the HTML and other resources in our database along with other information it needs to reproduce the sequence of events for this step during a load test. The visual rendering you see at the bottom of the step is an iFrame where our tool places a copy of what was received during the step creation process. It is NOT an accurate rendering as you would see it in a full browser without all of the other LoadStorm code and process
    “wrapped around” it. Essentially, what you are seeing is a page inside a page. The images may not display because of how they are referenced (e.g. absolute or relative) in the HTML code.

    The purpose of this iFrame display is to give the person building test scenarios confidence that the correct requests/responses were transacted for this step. The images you don’t see are still going to be requested during the load test, and all of the performance metrics such as response times will be accurate. Virtual users do not render the pages. No one will actually “see” the transactions during a load test because there is no need to see the pages, and our tool is measuring the Throughput, Error Rate, Response Times, Requests per Second, and Concurrent Users from many different load generation machines that process each virtual user going through the steps.

     

  • How do I stop a test that has killed my server?

    load testing stopYou can press the “STOP NOW” button at any time. It is found on the Run tab, and the button will only appear during the execution of a load test.

    Also, LoadStorm will automatically stop a load test if it can determine that your server has been overloaded. The thresholds for stopping a test are currently when the Error Rate exceeds 50% or the Average Response Time exceeds 15 seconds for the most recently summarized test period.

    Important Note for Small Tests: The sample interval is not the entire test but just the most recent part of it, and there must be a minimum number of requests in the sample. This interval is significant because in very small tests the number of failed requests AND the total requests may be few. For example, you must have some requests for at least 2 intervals, which means you have summarized more than 1 minute of requests.

     

  • Why does the Summary Table of test results show my missing JPGs in the HTML row?

    If one of your web pages contains a reference to an image that is not available on your system, then a response header will be sent to LoadStorm have Status Code = 404. Additionally, your web server will send an HTML page indicating the 404 “Not Found” page to alert the user. That’s standard practice in web applications.

    The reason LoadStorm reports the 404 errors as HTML is because your web server is sending our system the Content Type = “HTML” attribute. LoadStorm separates the data in the HTML and Other rows by the Content Type.

     

  • Does LoadStorm download images as a part of the load testing?

    Yes, and only if you want to. LoadStorm has a checkbox on the scenario edit page to turn download images on or off. There is a similar option for controlling the download and execution of Javascript files.

    Normally performance issues arise in the process of generating the dynamic page, and web servers can serve static images very quickly; however, we understand that many of our clients have websites with a large amount of media-related content, and load test results from those sites could be significantly affected by not downloading that content. Image download is especially relevant to performance if bottlenecks are in bandwidth or other network communications.

     

  • Why am I not getting as many page views as expected for the number of concurrent users in my test?

    Page views is a good term when reviewing Google Analytics data for traffic analysis. However, when reviewing load test data, page views can be a confusing metric. We do not track page views; rather we track requests and responses. Let’s look at the difference and how LoadStorm measures interactions with the server.

    The page views per virtual user should stay consistent as long as the response time is consistent and no errors. In some tests, the response time and errors are significant and increase over the duration of the test. For example, we see some load tests that have a large number of timeouts (no response to the request in 35 seconds). That means that LoadStorm cannot move to the next request for a virtual user until the response is received or we timeout the request. This drastically changes the equation you may be using to calculate the number of pageviews. Specifically, in one test there were 143,553 errors out of 1,345,534 total requests during a 30 minute test. This failure rate greatly decreased the number of requests LoadStorm was sending during the test.

    Also, if a request is sent for a step, LoadStorm is using the response header to capture the content type. A normal HTML page in a test returns, “text/html; charset=UTF-8″. When no response is received, LoadStorm assigns the response content type as, “NONE”. This affects the summary report table because the NONE responses are lumped into the OTHER category rather than the HTML category. Therefore, many of the 143k errors are shown in the OTHER category – not in the HTML category. That means that many of the HTML requests are not even shown in the table as HTML (or page views).

    The bottom line is that it’s mathematically invalid to expect linear correlation between page views, requests/responses, and vusers if the target server is not responding well. Everyone has a different definition of test success and failure. Some believe that greater than 1% error rate is a reason to stop the test. Others believe that an average response time greater than 3 seconds is a failure. Regardless how you define failure, timeouts and other server errors degrade the value of your load test metrics.

     


Pricing

  • How much does it cost?

    load testing costIt depends on how many simulated users you need for testing, and it depends on if you would like to use our consulting services.

    Creating an account is free, but the only limit of a free account is 50 concurrent users.

    For more details, please see Load Testing Cost.

  • Does LoadStorm use a full browser for accessing the websites to test or does it emulate a browser?

    load testing ramp timeLoadStorm emulates a browser. If we used an actual browser, we would trade-off the cost effectiveness of our solution for the ability to actually process exactly as a browser. Some solutions use actual browsers and cost you about 200-1,000% more per Virtual User Hour of testing.

    We understand some web developers need the real browsers and are willing to pay for it. Our load testing tool is designed to help those developers that achieve their load testing goals with emulation and a much lower cost.

    LoadStorm’s emulator makes the requests just like a browser, and it parses the HTML received in the response. As it parses the page, it identifies the additional resources needed such as images, stylesheets, XML, and javascript files. LoadStorm will automatically send requests to your server for these files. Each response is measured and recorded for speed and status codes.

    Our emulator will respect caching. For example, if you have the same logo image on every page, LoadStorm will only request it once for each virtual user, and subsequent steps will request any new images but not the ones already returned by your server.

    LoadStorm 2.0 effectively supports AJAX.

     

  • If I stop a test before my max users is reached, are the rest of my users wasted?

    Occasionally, a test will be terminated early because the target server fails or the tester stops it. It is not uncommon to have a test automatically stopped by LoadStorm when error rates hit 70% or so. When this happens in a test using virtual users, those users may seem “wasted”.

    Customers have asked us why the system deducted 5,000 VUsers from their account when the test only lasted 10 minutes and stopped at 500 users. The reason LoadStorm does not prorate the VUsers to the minute or to the actual peak users is because we incur the full costs of the 5,000 VUser test.

    Amazon’s cloud is elastic, and we “rent” the servers needed for your tests. However, there is a minimum of 1 hour increments that Amazon will bill us. For example, if your test needs 20 servers to hit your defined peak users, we buy those 20 servers upfront before your test actually starts. Then if you load test dies 5 minutes into it, we are still charged as if the test used all 20 servers for 1 hour.

    Thus, we recommend that you start with smaller tests to verify that you have all of the environmental factors properly configured. Growing the volume in several tests that increase in volume will potentially eliminate the “wasted” feeling of throwing 5,000 users at a server that fails at 500 users.

     


Running Load Tests

  • I can’t run a load test because a server isn’t verified or ignored. Can you help me?

    Manage Servers

    We can help with the verification process if you are stuck.  Please see our learning center for how to ‎Verify a Server.

    Another option is to Ignore the server. That simply means that LoadStorm will not send requests to it. By ignoring a server, your test may continue and generate load against your servers without making unnecessary requests to external sites such as Facebook, Twitter, or even Google Analytics.

    How does verification work?

    Our verification process works exactly like Google Analytics. The idea is to prove it is your server so LoadStorm isn’t used for a denial-of-service attack. You can put an empty file that is named with a unique string generated by LoadStorm (i.e. loadstorm-<unique string>) into your root directory (doesn’t matter what is in it). So let’s say your target server is http://www.xyzcorp.com, then there should be a valid response if a request comes to your server for http://www.xyzcorp.com/loadstorm-90630fc588.html.

    Alternatively, if you don’t want to use the file you can embed the code in your default page using comment tags. To continue the example, the HTML page received from requesting http://www.xyzcorp.com should contain a string within comment tags somewhere inside the document that matches “loadstorm-90630fc588″. LoadStorm will parse the page returned by your server exactly like a browser will. In that parsing, the verification string must appear somewhere for us to consider it verified.

    Example Code with a string in a comment tag:

    <html>
    <!– loadstorm-90630fc588 –>
    </html>

    Once you have the file or text in place, click on the Build… Servers tab, then select the the domain from the table that is your target server. Click the Check/Verify button under the verification section. If you are still having difficulties or need extra assistance, please contact support@loadstorm.com.

    Please only use the domain name when adding a server, and only use the domain name once. Adding a server for http://www.xyz.com works great and verification is successful. However, creating a server for http://www.xyz.com/myfolder will not work and trying to verify it will produce an error.

    Wonder why you need to verify servers?

    Please see our discussion on why servers need to be verified..

  • Why should I ramp up my test load volume over time?

    It helps you with performance engineering. We recommend that you do not begin a large test at the maximum number of virtual users. Rather than run a test for 10 minutes beginning with 5,000 and ending with 5,000, we suggest that you perhaps start with 500 users and increase to 5,000 over an hour. This approach has the advantage of allowing your system to get all the parts working properly at smaller load (e.g. caches, threads, database connections) before the heavier volume starts exposing potential bottlenecks in your application.

    From what we’ve seen in thousands of load tests with LoadStorm, it is common to see useful patterns in the metrics before the peak traffic is reached, and these metrics point to areas of performance limitations that can be tuned.

    Load testing is invariably coupled with performance tuning – an iterative process of test/tune/test/tune. Therefore, we recommend you ramp the volume in order to get more value from the test results.

     

  • Why doesn’t my test start immediately?

    LoadStorm needs a little time to prepare the test resources such as the load generation servers. Our system is using each of your scripts and steps to pre-calculate the virtual users’ actions, how many servers will need to be instantiated from the cloud, random think times, etc. This produces essentially a test roadmap that our Scheduler and Summarizer modules use to coordinate all the HTTP traffic and correctly capture the metrics. Depending on the number of concurrent users and the duration of your test, the preparation of a load test can take 2 to 15 minutes.

     

  • How does it generate HTTP traffic against my web application?

    LoadStorm is a truly distributed application that leverages the power of Amazon Web Services to scale on demand with processing power and bandwidth as needed to test the largest web projects. As you crank up the testing load from 200 to 150,000 virtual users, LoadStorm is automatically adding machines for you (as many as necessary) from Amazon’s server farm to handle the processing. When your tests are done and the extra machines are not needed, they are turned off to wait for another test.

    The normal processing is for a script to be processed as a sequential series of pages that you have defined in the recording. Once a page finishes, the VUser pauses for the specified time (between Min & Max “Think Time” in script), then the next page proceeds. After the last page completes, the VUser context (session, cache, cookies) is cleared and the script starts from the beginning again with a new VUser.

    You define the number of concurrent users you want to start with and peak at. As one VUser completes the script as described above, then another VUser is created to replace it – starting at page 1 again.

    If you have started at a small number and set the peak number much higher, then LoadStorm will not only retire VUsers that have completed the script, it will also add additional users throughout the test to ramp up the volume.
    This is repeated until the test is done as defined by the test duration field (in minutes) when you began the load test.

     

  • How Can I Increase the Requests per Second in My Test?

    Here are some thoughts on increasing RPS in LoadStorm tests:

    1. Reducing think time between pages in a script – minimum is 5 second pause, so set both min & max to 5.
    2. Increase the number of concurrent users.
    3. Reduce the response time from your server which decreases LoadStorm’s waiting for a response.

    RPS will always be impacted by the types of pages/resources your scripts are requesting. The way it works is that LoadStorm cranks up the starting concurrent users shortly after test starts. It then walks each user through the script assigned to it. One step is executed (such as home page), and all the images, JS, CSS, XML are requested separately. When those responses are all returned, then the system pauses a random time between the Min & Max “Think Time”, then it requests the next page. That continues until the end of the script, whereupon that virtual user is retired. After a few seconds, another vuser is created and the process starts again. So, having more users going at the same time, without waiting more than a few hundred ms to get a response, will increase RPS.

     

  • If I stop a test before my max users is reached, are the rest of my users wasted?

    Occasionally, a test will be terminated early because the target server fails or the tester stops it. It is not uncommon to have a test automatically stopped by LoadStorm when error rates hit 70% or so. When this happens in a test using virtual users, those users may seem “wasted”.

    Customers have asked us why the system deducted 5,000 VUsers from their account when the test only lasted 10 minutes and stopped at 500 users. The reason LoadStorm does not prorate the VUsers to the minute or to the actual peak users is because we incur the full costs of the 5,000 VUser test.

    Amazon’s cloud is elastic, and we “rent” the servers needed for your tests. However, there is a minimum of 1 hour increments that Amazon will bill us. For example, if your test needs 20 servers to hit your defined peak users, we buy those 20 servers upfront before your test actually starts. Then if you load test dies 5 minutes into it, we are still charged as if the test used all 20 servers for 1 hour.

    Thus, we recommend that you start with smaller tests to verify that you have all of the environmental factors properly configured. Growing the volume in several tests that increase in volume will potentially eliminate the “wasted” feeling of throwing 5,000 users at a server that fails at 500 users.

     

  • Why aren’t the 500 users evenly distributed across all 5 of my servers behind a round robin load balancer?

    A round robin load balancer will not work very well for testing your configuration unless you are using a much higher number of virtual users (at least 2000 or so). Each of our simulation servers will resolve the DNS name independently of the others, but each of our servers will simulate hundreds of users. I would expect a peak load of 500 users to get routed to more than one of your servers, but probably not all 5 of them. A peak load of 100 users or less is likely to hit only one of your servers.

     

  • Will the load test show up in the Google Analytics reports? How is testing typically tracked by Google Analytics?

    We would not expect LoadStorm test volume to show up in your Google Analytics. We do execute the javascript, but requests outside of your domain are blocked unless those other servers have also been verified. Servers like Google Analytics should be set to “Ignore”. So, the tracking by Analytics, ad servers, or other systems outside of your domain would not work. Some of these kinds of services have restrictions against load testing.

    Almost all of our customers don’t want our load testing tool traffic to show up in their marketing statistics (Google Analytics) because it essentially renders their analytics data worthless. Additionally, Google has limits on the amount of hits per period in their free GA account; otherwise you must upgrade to the paying account. There are also legal issues involved whereby Google doesn’t want automated tools like LoadStorm hammering their applications (even GA) – it is a violation of their standard terms and conditions. Thus, when a GA tracking javascript is found in a page that is involved in a LoadStorm test, that server is ignored automatically by our tool.

    Most of our clients use a different mechanism for watching the user activity of a load test. There are many tools for monitoring the website, including several excellent options for open source and commercial products such as New Relic, Loggly, and Hyperic. Most of these are very inexpensive or free, up to a certain level. These types of tools provide significant information regarding the user activity as well as important data on your architecture. For example, is your web server memory bound when response times start to increase drastically? Or did your database server hit 99% utilization when throughput drops?

    New Relic and Hyperic can monitor server activity from the local server using an agent. Loggly is a tool that has teamed up with New Relic for analyzing server logs which can make your life easier when trying to extrapolate the user activity from your server log.

    So to sum it up any javascript that reports traffic to third-party analytic tools will not work with LoadStorm. Instead useful tools for monitoring server usage and analyzing log files of user activity on the server under test (SUT) our encouraged and will work with LoadStorm.

  • Can your system run load tests behind my firewall for non-Internet applications?

    Unfortunately not. LoadStorm is only for web applications that can be reached from the Amazon cloud. If you have an intranet application, then opening a hole in your firewall will allow LoadStorm to run load tests against it.

     

  • How much does LoadStorm emulate the caching abilities of a real browser client? Will cache be reset across various sessions?

    The short answer is: If your browser caches it, LoadStorm will cache it for a virtual user. When a virtual user completes a script, it is retired which causes the session to end.

    Our load testing tool is designed to closely emulate the browser with a new, empty cache between sessions. So, generally your static assets will be cached for later pages in a script according to their header values. There are a few exceptions. We do not cache HTML resources since these are usually dynamically generated. Most LoadStorm users expect a subsequent HTML request to get this file again, thus that is how we coded it to work. If the headers are set properly, this will usually happen anyhow.

     

  • What happens to a virtual user when something fails or times out in a scenario?

    load testing vuserThe vuser continues to move to the next step in its scenario. For instance, if a JavaScript file fails to download during step 3:

    • That error will be logged in LoadStorm for reporting
    • That request will be terminated
    • All other requests for step 3 will be attempted as usual
    • The vuser will begin step 4 as defined in the scenario

    If any particular resource times out within a step, some of the step processing may not finish. This depends on what kind of resource fails and at what point in the step. If an image fails, this is relatively harmless. The images following will not be requested, but everything else will continue normally. If the html fails with a timeout, then the step will stop and no images or other resources will be requested. Images, javascript and css all depend on the html, so none will be requested for that step unless the html is received without error.

    The step following a step with an error will work normally to the extent that it can. As with a browser, you cannot click on a link or submit a form if you do not currently have a valid page available with which to work. You can always type in a new url and go from there. Likewise, in LoadStorm if the next step is an open step, then previous steps will have no effect except for possibly login or cookie state. If the next step is a click or form, then the previous html must have been received for the step to function. If the step cannot work because the previous html timed out or otherwise failed, then that step will show no requests or errors and the next step will be attempted after the expected pause.

    There is currently only one situation where a scenario will restart before all of its steps have run. This is a step timeout where a step starts but does not finish before 5 minutes have passed. A step includes getting all resources within the step such as html, javascript, images, css, etc. So, it is not inconceivable that this timeout occurs if there are 50, 100 or more resources in a step, but it is long enough that it is pretty rare.

     

  • What happens to virtual users as test scripts are running?

    LoadStorm creates the starting concurrent users within seconds of the test starting. For example, if you set the test to start at 1,000 users then you will have requests coming from those 1,000 users in the first 30 seconds. It then processes each virtual user through the script assigned to it. The first page, for example your home page, is execute first. The HTML is sent from your server and LoadStorm receives it. All other resources recorded (images, Javascript, CSS, and XML) are requested separately and sequentially.

    When all responses in a page are returned, the system pauses a random time between Min & Max Think Time pause (you can edit this), then it requests the next page. That continues until the virtual user reaches the end of the script, whereupon that VUser is “retired”. After about 30 seconds, another VUser is created and the process starts again. The cache for the new virtual user starts empty and does not remember cookies or static resources for the previous VUser.

     


Troubleshooting

  • I can’t run a load test because a server isn’t verified or ignored. Can you help me?

    Manage Servers

    We can help with the verification process if you are stuck.  Please see our learning center for how to ‎Verify a Server.

    Another option is to Ignore the server. That simply means that LoadStorm will not send requests to it. By ignoring a server, your test may continue and generate load against your servers without making unnecessary requests to external sites such as Facebook, Twitter, or even Google Analytics.

    How does verification work?

    Our verification process works exactly like Google Analytics. The idea is to prove it is your server so LoadStorm isn’t used for a denial-of-service attack. You can put an empty file that is named with a unique string generated by LoadStorm (i.e. loadstorm-<unique string>) into your root directory (doesn’t matter what is in it). So let’s say your target server is http://www.xyzcorp.com, then there should be a valid response if a request comes to your server for http://www.xyzcorp.com/loadstorm-90630fc588.html.

    Alternatively, if you don’t want to use the file you can embed the code in your default page using comment tags. To continue the example, the HTML page received from requesting http://www.xyzcorp.com should contain a string within comment tags somewhere inside the document that matches “loadstorm-90630fc588″. LoadStorm will parse the page returned by your server exactly like a browser will. In that parsing, the verification string must appear somewhere for us to consider it verified.

    Example Code with a string in a comment tag:

    <html>
    <!– loadstorm-90630fc588 –>
    </html>

    Once you have the file or text in place, click on the Build… Servers tab, then select the the domain from the table that is your target server. Click the Check/Verify button under the verification section. If you are still having difficulties or need extra assistance, please contact support@loadstorm.com.

    Please only use the domain name when adding a server, and only use the domain name once. Adding a server for http://www.xyz.com works great and verification is successful. However, creating a server for http://www.xyz.com/myfolder will not work and trying to verify it will produce an error.

    Wonder why you need to verify servers?

    Please see our discussion on why servers need to be verified..

  • Why do I need to verify my server? How do I do that?

    This is a very common question. In fact, a good example is the email this week from a new user stating, “Just tried to test your service and it came back with these comments:

    There were problems with the following fields:

    I am not sure what to do now…Why do I have to verify these servers?”

    Are You Load Testing Facebook?

    However, the error messages above indicate that his test plan will be hitting servers that aren’t verified (authorized for testing). Put another way, his scenarios contain requests that go to servers that aren’t his, and LoadStorm gets in trouble when our customers load test other people’s servers without their permission.

    Does he really want to hit Twitter and Facebook for his load test? If so, is he testing their ability to handle his load? If so, does he have their permission to hammer on their servers? Probably not. They tend to get upset when someone utilizes a cloud load testing tool to simulate thousands of VUsers against their social media platform. Please read the terms and conditions of use for those services.

    And where do those VUsers originate? LoadStorm. So who gets the phone call from their lawyers? We do.

    Prevent Denial of Service Attacks

    Our verification process is designed to prevent LoadStorm from being used as a Denial of Service (DOS) weapon. By placing a code on your home page that LoadStorm can request and confirm, our system confirms that you have the control over that server; thus, it is safe for our tool to hammer it with VUsers.

     

  • Does LoadStorm use a full browser for accessing the websites to test or does it emulate a browser?

    load testing ramp timeLoadStorm emulates a browser. If we used an actual browser, we would trade-off the cost effectiveness of our solution for the ability to actually process exactly as a browser. Some solutions use actual browsers and cost you about 200-1,000% more per Virtual User Hour of testing.

    We understand some web developers need the real browsers and are willing to pay for it. Our load testing tool is designed to help those developers that achieve their load testing goals with emulation and a much lower cost.

    LoadStorm’s emulator makes the requests just like a browser, and it parses the HTML received in the response. As it parses the page, it identifies the additional resources needed such as images, stylesheets, XML, and javascript files. LoadStorm will automatically send requests to your server for these files. Each response is measured and recorded for speed and status codes.

    Our emulator will respect caching. For example, if you have the same logo image on every page, LoadStorm will only request it once for each virtual user, and subsequent steps will request any new images but not the ones already returned by your server.

    LoadStorm 2.0 effectively supports AJAX.

     

  • What are the IP address ranges used by LoadStorm? We need to white list them.

    Because we use the Amazon EC2 cloud to dynamically instantiate load generation servers, we don’t have a specific static IP address, but here are the ranges:

    US East (Northern Virginia):

    72.44.32.0/19 (72.44.32.0 – 72.44.63.255)
    67.202.0.0/18 (67.202.0.0 – 67.202.63.255)
    75.101.128.0/17 (75.101.128.0 – 75.101.255.255)
    174.129.0.0/16 (174.129.0.0 – 174.129.255.255)
    204.236.192.0/18 (204.236.192.0 – 204.236.255.255)
    184.73.0.0/16 (184.73.0.0 – 184.73.255.255)
    184.72.128.0/17 (184.72.128.0 – 184.72.255.255)
    184.72.64.0/18 (184.72.64.0 – 184.72.127.255)
    50.16.0.0/15 (50.16.0.0 – 50.17.255.255)
    50.19.0.0/16 (50.19.0.0 – 50.19.255.255)
    107.20.0.0/14 (107.20.0.0 – 107.23.255.255)
    23.20.0.0/14 (23.20.0.0 – 23.23.255.255)
    54.242.0.0/15 (54.242.0.0 – 54.243.255.255)
    54.234.0.0/15 (54.234.0.0 – 54.235.255.255)
    54.236.0.0/15 (54.236.0.0 – 54.237.255.255)
    54.224.0.0/15 (54.224.0.0 – 54.225.255.255)
    54.226.0.0/15 (54.226.0.0 – 54.227.255.255)
    54.208.0.0/15 (54.208.0.0 – 54.209.255.255)
    54.210.0.0/15 (54.210.0.0 – 54.211.255.255)
    54.221.0.0/16 (54.221.0.0 – 54.221.255.255)
    54.204.0.0/15 (54.204.0.0 – 54.205.255.255)
    54.196.0.0/15 (54.196.0.0 – 54.197.255.255)
    54.198.0.0/16 (54.198.0.0 – 54.198.255.255)
    54.80.0.0/13 (54.80.0.0 – 54.87.255.255)
    54.88.0.0/14 (54.88.0.0 – 54.91.255.255)
    54.92.0.0/16 (54.92.0.0 – 54.92.255.255)
    54.92.128.0/17 (54.92.128.0 – 54.92.255.255)

    US West (Oregon):

    50.112.0.0/16 (50.112.0.0 – 50.112.255.255)
    54.245.0.0/16 (54.245.0.0 – 54.245.255.255)
    54.244.0.0/16 (54.244.0.0 – 54.244.255.255)
    54.214.0.0/16 (54.214.0.0 – 54.214.255.255)
    54.212.0.0/15 (54.212.0.0 – 54.213.255.255)
    54.218.0.0/16 (54.218.0.0 – 54.218.255.255)
    54.200.0.0/15 (54.200.0.0 – 54.201.255.255)
    54.202.0.0/15 (54.202.0.0 – 54.203.255.255)
    54.184.0.0/13 (54.184.0.0 – 54.191.255.255)

    US West (Northern California):

    204.236.128.0/18 (204.236.128.0 – 204.236.191.255)
    184.72.0.0/18 (184.72.0.0 – 184.72.63.255)
    50.18.0.0/16 (50.18.0.0 – 50.18.255.255)
    184.169.128.0/17 (184.160.128.0 – 184.169.255.255)
    54.241.0.0/16 (54.241.0.0 – 54.241.255.255)
    54.215.0.0/16 (54.215.0.0 – 54.215.255.255)
    54.219.0.0/16 (54.219.0.0 – 54.219.255.255)
    54.193.0.0/16 (54.193.0.0 – 54.193.255.255)
    54.176.0.0/15 (54.176.0.0 – 54.177.255.255)
    54.183.0.0/16 (54.183.0.0 – 54.183.255.255)

    EU (Ireland):

    79.125.0.0/17 (79.125.0.0 – 79.125.127.255)
    46.51.128.0/18 (46.51.128.0 – 46.51.191.255)
    46.51.192.0/20 (46.51.192.0 – 46.51.207.255)
    46.137.0.0/17 (46.137.0.0 – 46.137.127.255)
    46.137.128.0/18 (46.137.128.0 – 46.137.191.255)
    176.34.128.0/17 (176.34.128.0 – 176.34.255.255)
    176.34.64.0/18 (176.34.64.0 – 176.34.127.255)
    54.247.0.0/16 (54.247.0.0 – 54.247.255.255)
    54.246.0.0/16 (54.246.0.0 – 54.246.255.255)
    54.228.0.0/16 (54.228.0.0 – 54.228.255.255)
    54.216.0.0/15 (54.216.0.0 – 54.217.255.255)
    54.229.0.0/16 (54.229.0.0 – 54.229.255.255)
    54.220.0.0/16 (54.220.0.0 – 54.220.255.255)
    54.194.0.0/15 (54.194.0.0 – 54.195.255.255)
    54.72.0.0/14 (54.72.0.0 – 54.75.255.255)
    54.76.0.0/15 (54.76.0.0 – 54.77.255.255)
    54.78.0.0/16 (54.78.0.0 – 54.78.255.255)
    54.74.0.0/15 (54.74.0.0 – 54.75.255.255)
    185.48.120.0/22 (185.48.120.0 – 185.48.123.255)

    Asia Pacific (Singapore)

    175.41.128.0/18 (175.41.128.0 – 175.41.191.255)
    122.248.192.0/18 (122.248.192.0 – 122.248.255.255)
    46.137.192.0/18 (46.137.192.0 – 46.137.255.255)
    46.51.216.0/21 (46.51.216.0 – 46.51.223.255)
    54.251.0.0/16 (54.251.0.0 – 54.251.255.255)
    54.254.0.0/16 (54.254.0.0 – 54.254.255.255)
    54.255.0.0/16 (54.255.0.0 – 54.255.255.255)
    54.179.0.0/16 (54.179.0.0 – 54.179.255.255)

    Asia Pacific (Sydney)

    54.252.0.0/16 (54.252.0.0 – 54.252.255.255)
    54.253.0.0/16 (54.253.0.0 – 54.253.255.255)
    54.206.0.0/16 (54.206.0.0 – 54.206.255.255)
    54.79.0.0/16 (54.79.0.0 – 54.79.255.255)

    Asia Pacific (Tokyo)

    175.41.192.0/18 (175.41.192.0 – 175.41.255.255)
    46.51.224.0/19 (46.51.224.0 – 46.51.255.255)
    176.32.64.0/19 (176.32.64.0 – 176.32.95.255)
    103.4.8.0/21 (103.4.8.0 – 103.4.15.255)
    176.34.0.0/18 (176.34.0.0 – 176.34.63.255)
    54.248.0.0/15 (54.248.0.0 – 54.249.255.255)
    54.250.0.0/16 (54.250.0.0 – 54.250.255.255)
    54.238.0.0/16 (54.238.0.0 – 54.238.255.255)
    54.199.0.0/16 (54.199.0.0 – 54.199.255.255)
    54.178.0.0/16 (54.178.0.0 – 54.178.255.255)
    54.95.0.0/16 (54.95.0.0-54.95.255.255)
    54.92.0.0/17 (54.92.0.0 – 54.92.127.255)

     
    We obtained these ranges from Amazon’s EC2 list of public IP ranges.

  • Other than IP address, is there any way to identify LoadStorm requests from normal traffic?

    Yes, there are two additional ways for you to white list our traffic:

    1. The user agent in the HTTP header includes LoadStorm. However, if your test scripts contain steps with HTTPS requests, then those headers will be encrypted.
    2. If you are not testing against your production server, we recommend trying a non-standard port number to identify the traffic as load testing. LoadStorm allows you to use any port number and can be designated for each target server.

    Thumbs Up Button

  • Can I create a test script where virtual users login to my web application? If so, how do I get my test user file into LoadStorm?

    Yes. Once you’ve created a script, you may modify forms in order to log into your application with test user data.

    We recommend you visit our learning center, where we walk you through the parameterization process for a script. A brief explanation is given below.

    While creating a recording using developer tools in your browser, log in to your application. Save the recording as a HAR file and upload it into your account. Open your script and select the parameterization tab.

    After that, upload your CSV which contains necessary login info (username/email, passwords) into the Build… User Data section of LoadStorm. Note that the first row’s data will be read as column headers and not row data. You can also choose to “generate data” on-the-fly in LoadStorm. If you need to embed a comma inside a particular data field, simply place quotation marks around all field data in the file.

    Finally, edit the necessary script and find the request that submits the form. Modify the form to include test user information. To add the test user data, you must select “custom” under the form modifications options, and select the data from a CSV file.

    This simulates traffic more realistically. Instead of simulating one user logging in over and over, many different users log in instead.

     

  • Can your system run load tests behind my firewall for non-Internet applications?

    Unfortunately not. LoadStorm is only for web applications that can be reached from the Amazon cloud. If you have an intranet application, then opening a hole in your firewall will allow LoadStorm to run load tests against it.

     

  • What happens to a virtual user when something fails or times out in a scenario?

    load testing vuserThe vuser continues to move to the next step in its scenario. For instance, if a JavaScript file fails to download during step 3:

    • That error will be logged in LoadStorm for reporting
    • That request will be terminated
    • All other requests for step 3 will be attempted as usual
    • The vuser will begin step 4 as defined in the scenario

    If any particular resource times out within a step, some of the step processing may not finish. This depends on what kind of resource fails and at what point in the step. If an image fails, this is relatively harmless. The images following will not be requested, but everything else will continue normally. If the html fails with a timeout, then the step will stop and no images or other resources will be requested. Images, javascript and css all depend on the html, so none will be requested for that step unless the html is received without error.

    The step following a step with an error will work normally to the extent that it can. As with a browser, you cannot click on a link or submit a form if you do not currently have a valid page available with which to work. You can always type in a new url and go from there. Likewise, in LoadStorm if the next step is an open step, then previous steps will have no effect except for possibly login or cookie state. If the next step is a click or form, then the previous html must have been received for the step to function. If the step cannot work because the previous html timed out or otherwise failed, then that step will show no requests or errors and the next step will be attempted after the expected pause.

    There is currently only one situation where a scenario will restart before all of its steps have run. This is a step timeout where a step starts but does not finish before 5 minutes have passed. A step includes getting all resources within the step such as html, javascript, images, css, etc. So, it is not inconceivable that this timeout occurs if there are 50, 100 or more resources in a step, but it is long enough that it is pretty rare.