All FAQ

  • I can’t run a load test because a server isn’t verified or ignored. Can you help me?

    Manage Servers

    Reach out to support@loadstorm.com for help with the verification process if you are stuck. For instructions on how to verify a server, please see our learning center‎ or watch this video tutorial.

    How does verification work?

    Our verification process works like Google Analytics. The idea is to prove it is your server so LoadStorm isn’t used for a denial-of-service attack. You can put an empty file that is named with a unique string generated by LoadStorm (i.e. loadstorm-<unique string>) into your root directory (doesn’t matter what is in it). So let’s say your target server is http://www.xyzcorp.com, then there should be a valid response if a request comes to your server for http://www.xyzcorp.com/loadstorm-90630fc588.html.

    Alternatively, if you don’t want to use the file you can embed the code in your default page using HTML comment tags. To continue the example, the HTML page received from requesting http://www.xyzcorp.com should contain a string within comment tags somewhere inside the document that matches “loadstorm-90630fc588″. LoadStorm will parse the page returned by your server exactly like a browser will. In that parsing, the verification string must appear somewhere for us to consider it verified.

    Example Code with a string in a comment tag:
    <html>
    <!– loadstorm-90630fc588 –>
    </html>

  • What does LoadStorm do?

    load testing info
    LoadStorm™ is a web-based load testing tool for simulating what users do with a web site or web application. You use it to build tests that send requests to your server in the same way that a user’s browser sends requests to your server. But these tests are executed by our automated systems rather than by a user, so they can be done repeatedly and in large numbers simultaneously. They can also be built using our tool in such a way as to simulate a large number of different users with different tasks to perform.

     

  • How much does it cost?

    load testing cost
    It depends on how many virtual users you need for testing, and it depends on if you would like to use our consulting services.

    Creating an account is free, but a free account is limited to 50 virtual users to try out our product. By default, we offer four pricing plans. Each plan scales up in cost, but offers more benefits such as bundled consulting hours.

    For more details, please see Load Testing Cost.

     

  • Why do I need to verify my server? How do I do that?

    This is a very common question. A example email may look something like this:

    I am unable to schedule a test run because LoadStorm shows that I need to verify a bunch of servers:

    which-servers-need-to-be-verified

    Click to zoom.

    I am not sure what to do now…Why do I have to verify these servers?

    Sincerely, Mr. WPL

    Are You Load Testing Facebook?

    From the selected script LoadStorm found that this test plan will be hitting servers that aren’t verified (authorized for testing). Put another way, these scripts contain requests that go to servers that do not belong to Mr. WPL in the example email, and LoadStorm gets in trouble when our customers load test other people’s servers without their permission.

    Does Mr. WPL really want to hit Facebook and Pinterest as part of his load test? If so, do those 3rd party servers contribute a crucial part of his server’s ability to handle load? If so, does he have their permission to hammer on their servers? Probably not. They tend to get upset when someone utilizes a cloud load testing tool to simulate thousands of VUsers against their social media platform. Please read the terms and conditions of use for those services.

    And where do those VUsers originate? LoadStorm. So who gets the phone call from their lawyers? We do.

    Prevent Denial of Service Attacks

    Our verification process is designed to prevent LoadStorm from being used as a Denial of Service (DOS) weapon. By placing a code on your home page that LoadStorm can request and confirm, our system confirms that you have the control over that server; thus, it is safe for our tool to hammer it with VUsers.

    Recommendation

    To verify a server follow the instructions in our learning center or watch this video tutorial.

    Ignore all third-party servers that are not crucial to your web application. LoadStorm will not make any requests to ignored servers. Remember the goal is to test the scalability of your web servers, and not those of third-parties. If you feel there is a third-party server that is crucial to your testing needs please email them and copy support@loadstorm.com requesting permission from their company to let you use our software to load test their web servers and yours as part of your testing needs.

  • How should I set up scripts to simulate real traffic?

    Most people create several scripts to accurately reflect different types of users. Scripts execute simultaneously in order to hit your site with the correct mix of traffic. You control the weighting of each script by giving it a number relative to the others. For example, if you have 2 scripts and you give one a weighting of 4 and the other a weighting of 1, then the first will generate 80% of the traffic and the second 20%. This 80/20 could be realistic for a retail e-commerce site where most of the people (80%) are browsing the product catalog, while a minority of people (20%) are going through a buying experience. You can of course throw in a low-weight scenario to represent some internal employees adding products or issuing refunds.

    Another way of entering the weight of the traffic is to just enter the exact percentage you would like each script to carry. For the example above you would simply enter 80 and 20. The only thing you have to be very careful about is that the percentages add up to 100. Otherwise, it will work like the above description, which is different.

    There is no limit to the number of scripts you can create, and you can use many scripts in a test. We encourage you to build as many as necessary to accurately reflect the real traffic patterns on your site. That will give you the most meaningful metrics for tuning your applications.

    We also simulate the user “think time” with random pauses after each page. Real users click on something, then read the page or fill in a form, and then click on something else. Thus, you don’t want to have a simulated virtual user clicking on something every .5 seconds – it would not give you meaningful results if you want to see how your site holds up in the real world.

    Also see: Virtual User Calculations about calculating the amount of virtual users needed for a realistic test.

  • Why should I ramp up my test load volume over time?

    It helps with performance engineering. We recommend to not begin a large test at the maximum number of virtual users. Rather than run a test for 10 minutes beginning with 5,000 and ending with 5,000, we suggest to start smaller, for example, with 500 users and increase to 5,000 over an hour. This approach has the advantage of allowing your system to get all the parts working properly at smaller load (e.g. caches, threads, database connections) before the heavier volume starts exposing potential bottlenecks in your application. Also if the target server begins to fail during the test this provides more granularity into the number of concurrent users where failure begins. If the test starts at 5,000 and fails all that can be learned from the results is that it fails somewhere between 1 and 5,000 which isn’t very useful.

    From what we’ve seen in thousands of load tests with LoadStorm, it is common to see useful patterns in the metrics before the peak traffic is reached, and these metrics point to areas of performance limitations that can be tuned.

    Load testing is invariably coupled with performance tuning – an iterative process of test/tune/test/tune. Therefore, we recommend you ramp the volume in order to get more value from the test results.

     

  • Why doesn’t my test start immediately?

    LoadStorm needs a little time to prepare the test resources such as the load generation servers. Our system is using each selected script(s) to pre-calculate the virtual users’ actions, how many servers will need to be instantiated from the cloud, random think times, etc. This produces essentially a test roadmap that our Scheduler and Summarizer modules use to coordinate all the HTTP traffic and correctly capture the metrics. Depending on the number of concurrent users and the duration of your test, the preparation of a load test can take 2 to 15 minutes.

     

  • Does LoadStorm use a full browser for accessing the websites to test or does it emulate a browser?

    load testing ramp time
    LoadStorm PRO does not use a browser emulator, nor is it a full browser. Instead, it takes a HAR recording and builds a script from the recorded network requests to mimic a real browser while still allowing for a competitively costed solution. LoadStorm PRO also supports AJAX requests as well as REST and SOAP API calls. If we used an actual browser, we would trade-off the cost effectiveness of our solution for the ability to actually process javascript and other interactions exactly as a browser. Some solutions use actual browsers and cost you a great deal more money.

    We understand some web developers need real browsers and are willing to pay for it. Our load testing tool is designed to help those developers that are willing to achieve their load testing goals without a full browser and at a much lower cost.

     

  • What are the IP address ranges used by LoadStorm? We need to white list them.

    Because we use the Amazon EC2 cloud to dynamically instantiate load generation servers we don’t have a specific static IP address. However, we can provide the possible ranges for each EC2 data center or geographic location that we use. The only groups of ranges that we do not use are GovCloud, and Beijing. The list of ranges are no longer updated from Amazon’s blog post about EC2 public IP ranges.

    Amazon now maintains the list of their EC2 public IP ranges in this JSON file:
    https://ip-ranges.amazonaws.com/ip-ranges.json

  • Other than IP address, is there any way to identify LoadStorm requests from normal traffic?

    Yes, there are two additional ways for you to white list our traffic:

    1. You can customize the user agent for each script which your firewall can see in the request headers of inbound requests. However, if your test scripts contains HTTPS requests, then those headers will be encrypted. Visit our learning center for more information on modifying a script’s default settings such as the User Agent.
    2. If you are not testing against your production server, we recommend trying a non-standard port number to identify the traffic as load testing. LoadStorm allows you to use any port number and can be designated for each target server.

    Thumbs Up Button

  • When configuring a scenario step that clicks a link, how does LoadStorm follow the link?

    LoadStorm follows links when the session ID is appended to the URL query string because it searches for the text between the open and close anchor tags. If it doesn’t find that text, then the fall back is to match the XPATH. We stored the XPATH when the scenario was created (essentially an address inside the page’s DOM). The XPATH is a unique identifier that allows us to click on the anchor such as an image or the href attribute or execute the Javascript.

     

  • What is the “Average Duration” for a scenario?

    Average Duration is the time taken for ALL steps to complete (see step duration calculation), plus the average pause times between steps. Thus, it is the mean time for a specific vuser’s “life”. The actual duration fluctuates with the amount of pause assigned to that vuser. The Average Duration is shown above the steps on the Scenario page.

     

  • How is “Duration” for a scenario step calculated?

    Duration is the time taken for a scenario step to complete, which is the sum of time to last byte for each request in the step (including HTML, JS, CSS, images, etc.). The Average Duration shown above the steps on the Scenario page is the total of all step durations plus the average pause times between steps.

     

  • What does “response time” mean?

    load testing info
    LoadStorm calculates the response time in the following manner:

    The total time taken between the first byte of the request sent and the last byte of the response received for all requests made. This includes the network latency and the time for the server to respond.

    The primary page normally includes the HTML, CSS, and Javascript files. However, the primary page could be another resource such as a PDF, video, or Flash file. The response time may appear slightly faster than observed by you in a browser because it does not include time for a browser to render the page. Of course render time will be affected by browser type, operating system, and horsepower of the client computer.

    It is also important to note that LoadStorm’s servers probably have faster internet connections than many homes and offices.

     

  • Can I create a test script where virtual users login to my web application? If so, how do I get my test user file into LoadStorm?

    Yes. Once you’ve created a script, you may modify form POSTs in order to log into your application with test user data.

    We recommend you visit our learning center, where we go over all of the options for parameterization in a script. A brief explanation is given below, but we also offer video tutorials on common examples such as parameterizing a login form.

    While creating a recording using developer tools in your browser, log into your application, and save the recording as a HAR file. Then upload the recording into your LoadStorm account. Open your script and select the parameterization tab.

    After that, click BUILD in the left navigation and switch to the User Data tab at the top. Here you can upload your CSV which contains necessary login info (username/email, passwords). Note that the first row’s data will be read as column headers and not row data. You can also choose to “generate data” on-the-fly in LoadStorm. If you need to embed a comma inside a particular data field, simply place quotation marks around all field data in the file.

    Finally, edit the necessary script and find the POST request that submits the form. Modify the form to include Custom Data with CSV user information. To add the test user data, you must select “custom” under the form modifications options, and select the data from a CSV file.

    This simulates traffic more realistically. Instead of simulating one user logging in over and over, many different users log in instead.

  • How does it generate HTTP traffic against my web application?

    LoadStorm is a truly distributed application that leverages the power of Amazon Web Services to scale on demand with processing power and bandwidth as needed to test the largest web projects. As you crank up the testing load from 200 to 1,000,000 virtual users, LoadStorm is automatically adding machines for you (as many as necessary) from Amazon’s data centers to handle the processing. When your tests are done and the extra machines are not needed, they are turned off to wait for another test.

    A script is processed as a sequential series of pages that you have defined in the recording. Once a page finishes, the VUser pauses for the specified time (between Min & Max “Think Time” in the script), then the next page proceeds. After the last page completes, the VUser context (session, cache, cookies) is cleared and the script starts from the beginning again with the same VUser. This same VUser will use different data upon script repeat if a CSV of User Data or dynamic response text has been parameterized within the script. As the concurrent users scale up they each begin the script for the first time, and also repeat the script after each completion.

    You define the number of concurrent users you want at the start and peak of the test. You can also define a ramp down period to see how well your system recovers as the traffic subsides.

    Scripts are repeated until the test is done as defined by the test duration field (in minutes) from when you began the load test.

     

  • What is the “Form data set” option when creating a new scenario? Why are there options for Tiny, Small, and Huge?

    The Form Data Set field that you see on the Scenario settings page is a mechanism to select an input source for form submission. Using the Form Data Sets functionality, it is possible to submit forms such as a login form and fill-in the fields (e.g. username, password) with data from your system. Other common types of form submission are search terms, credit card numbers, and profile preferences. Any type of form field that can be filled in with text can utilize the Form Data Sets.

    You may upload your own data to be used for form submission, or you may use one of our built-in form data sets.

    The Tiny, Small, and Huge form data sets are built-into LoadStorm. The only real difference between the three files are the number of rows. Tiny only has 10 records. Small has 100, and Huge has 10,000 records of user information. They contain user information:

    • First Name
    • Last Name
    • Username
    • Password
    • Email address

    Many of our customers use one of our built-in form data sets for registration on their web application or for logging in virtual users.

    If you are going to upload your own data into LoadStorm, here is an example of the process for getting user credentials from your system to LoadStorm:

    1. Export a CSV file of user information from your user table(s).
    2. The first row of the CSV must contain the names of the columns, and these column names will be mapped to the field name in the step that submits the form.
    3. The CSV file is uploaded by clicking on the UPLOAD DATA button on the Build tab. This button will be near the bottom, under the Form Data Sets section of the page.
    4. You must give the file a label so that you can choose that file to be used at the scenario level.

    Once you have the CSV data in LoadStorm (you should see it on the Build tab under the heading Form Data Sets), then you need to edit the scenario and choose the appropriate data set based on the label you assigned when uploading it. This essentially connects the CSV file to your scenario. Then when you add a step for form submission, you will be able to map the fields in the form to the columns of data in the Form Data Set.

    The LoadStorm Tour has a video tutorial showing how to create a login form submission. Watch the video on YouTube

     

  • How Can I Increase the Requests per Second in My Test?

    Here are some thoughts on increasing RPS in LoadStorm tests:

    1. Reducing the think time added after each page in a script; the minimum pause is 5 seconds, so set both min & max to 5 seconds.
    2. Increasing the number of concurrent users.
    3. Reducing the response time from your server which decreases the time LoadStorm has to wait for a request to complete.
    4. Compressing the size of the response is one way to decrease the time it takes for the request to complete as well as save on throughput.

    RPS will always be impacted by the types of pages/resources your scripts are requesting. The way it works is that LoadStorm cranks up the starting concurrent users shortly after test starts. It then walks each user through the script assigned to it. The GET requests will be made in asynchronous bursts of up to 6 at a time such as HTML, images, JS, CSS, and XML. Each POST request will be made on its own to ensure that any dynamic response text from earlier requests was available for to be passed into the POST data. This also ensures that any response data from the POST is available for subsequent requests. When those responses are all returned, then the system pauses a random time between the Min & Max “Think Time”, then it requests the next page. This process continues until the end of the script. So, having more users going at the same time, without waiting more than a few hundred ms to get a response, will increase RPS.

     

  • Do you provide a URL import functionality? I need to perform a test that includes roughly 100,000 URLs, however I do not want to have to enter in each URL individually.

    The concept is to upload a file of strings that can be substituted in a single step of a scenario. Most customers use this functionality to append a query string on the end of a domain. You can produce a CSV with one or more columns containing the strings. Then you add a step that opens a page, and there is specific syntax for mapping the string from a column of the CSV into the URL requested by the step.

    Here’s an example of what would be used for the step:

    http://cia-factbook.s3.amazonaws.com#{FileOfPaths.pathFull1}

    The hash and open brace are significant because they tell LoadStorm to substitute the text. “FileOfPaths” is the label for the CSV I uploaded into my LoadStorm account. “pathFull1″ is the column name inside FileOfPaths that contains the string I want to append on the domain.

    To upload the CSV (form data set) of your URLs/query strings into LoadStorm:

    1. Export a CSV file with one or more columns containing the URLs.
    2. The first row of the CSV must contain the names of the columns, and these column names will be mapped to the substitution point in the step indicated by the #{ } delimiters.
    3. The CSV file is uploaded by clicking on the UPLOAD DATA button on the Build tab. This button will be near the bottom, under the Form Data Sets section of the page.
    4. You must give the file a label so that you can choose that file to be used at the scenario level.

    Once you have the CSV data in LoadStorm (you should see it on the Build tab under the heading Form Data Sets), then you need to edit the scenario and choose the appropriate data set based on the label you assigned when uploading it. This essentially connects the CSV file to your scenario. Then when you add a step for using the URL/query string, you will be able to place #{formDataSet.columnName} from the Form Data Set.

     

  • Why does my step not show my images or contents aren’t displaying properly?

    When adding a step in LoadStorm, our emulator makes the request to your web application and receives the response containing the HTML code for the page. Our tool then parses the HTML to determine what additional resources (e.g. images, Javascript libraries) need to be requested for that page.

    Each of those resources are requested and received. LoadStorm saves all of the HTML and other resources in our database along with other information it needs to reproduce the sequence of events for this step during a load test. The visual rendering you see at the bottom of the step is an iFrame where our tool places a copy of what was received during the step creation process. It is NOT an accurate rendering as you would see it in a full browser without all of the other LoadStorm code and process “wrapped around” it. Essentially, what you are seeing is a page inside a page. The images may not display because of how they are referenced (e.g. absolute or relative) in the HTML code.

    The purpose of this iFrame display is to give the person building test scenarios confidence that the correct requests/responses were transacted for this step. The images you don’t see are still going to be requested during the load test, and all of the performance metrics such as response times will be accurate. Virtual users do not render the pages. No one will actually “see” the transactions during a load test because there is no need to see the pages, and our tool is measuring the Throughput, Error Rate, Response Times, Requests per Second, and Concurrent Users from many different load generation machines that process each virtual user going through the steps.

     

  • Can I change the host file for my test? I need to hit a server that does NOT currently have a valid DNS name because we don’t want it found by customers until we launch the site.

    Unfortunately, our load testing tool doesn’t have a feature for what you need. Normally this is a simple process of putting a domain name to IP address mapping in a configuration file on a particular machine. The process is not as easy in our situation because we use dynamically instantiated servers from the Amazon cloud; thus, we do not know what IP addresses we will be using before you run your tests. We would need to create new images specifically for your tests, and we would need to make changes to our core launching software that instantiates the images so that new machines are launched with this particular host mapping. If you would like us to coordinate this custom process, please contact our support. Additional charges per test will apply.

     

  • If I stop a test before reaching the peak users, are the rest of my users wasted?

    You can terminate a test early because the target server fails. When this happens in a test using virtual users, those users may seem “wasted”.

    Customers have asked us why the system deducted 5,000 VUsers from their account when the test only lasted 10 minutes and stopped at 500 users. The reason LoadStorm does not prorate the VUsers to the minute or to the actual peak users is because we incur the full costs of the 5,000 VUser test.

    Amazon’s cloud is elastic, and we “rent” the servers needed for your tests. However, there is a minimum of 1 hour increments that Amazon will bill us. For example, if your test needs 20 servers to hit your defined peak users, we buy those 20 servers upfront before your test actually starts. Then if your load test dies 5 minutes into it, we are still charged as if the test used all 20 servers for 1 hour.

    Thus, we recommend that you start with smaller tests to verify that you have all of the environmental factors properly configured. Growing the volume in several tests that increase in volume will potentially eliminate the “wasted” feeling of throwing 5,000 users at a server that fails at 500 users.

     

  • Why not let our users test the system’s capacity?

    For some projects, this is an acceptable approach. For some internal applications, apps that are very simple will realistically expect light traffic. The best plan may be to get it done as quickly as possible. You have to ask yourself what is the risk of the application not working at some point. For many businesses, getting a large influx of new users is the peak of success, and that success would be shattered by having the new users unable to use the system, unable to buy your product and unhappy with your company.

     

  • How do I stop a test that has killed my server?

    You can press the “Stop Execution” button during a load test. To do this:

    1. click ANALYZE in the left navigation
    2. double-click the test run that is currently in progress
    3. near the top-right in the filters area is the “Stop Execution” button

    How to stop a load test in progress
     

  • Why does the Summary Table of test results show my missing JPGs in the HTML row?

    If one of your web pages contains a reference to an image that is not available on your system, then a response header will be sent to LoadStorm with Status Code = 404. Additionally, your web server will send an HTML page indicating the 404 “Not Found” page to alert the user. That’s standard practice in web applications.

    The reason LoadStorm reports the 404 errors as HTML is because your web server is sending our system the Content Type = “HTML” attribute. LoadStorm separates the data by the Content Type.

     

  • With a monthly subscription, are you able to add additional one-time VUsers?

    Yes, you may add Storm on Demand users for any load test. This allows you to spike test traffic to a much higher level than your subscription.

    For example, if you have a Hurricane subscription for 5,000 maximum concurrent vusers, you can purchase an additional 10,000 Storm on Demand users for a total of 15,000 vusers on a single test. Those 10,000 on demand users would be available for only one load test, and the 5,000 are available for other tests throughout the month.

     

  • Why aren’t the 500 users evenly distributed across all 5 of my servers behind a round robin load balancer?

    A round robin load balancer will not work very well for testing your configuration unless you are using a much higher number of virtual users (at least 1000 or more). Each of our simulation servers will resolve the DNS name independently of the others, but each of our servers will simulate hundreds of users. I would expect a peak load of 500 users to get routed to more than one of your servers, but probably not all 5 of them. A peak load of 200 users or less is likely to hit only one of your servers.

     

  • Will the load test show up in the Google Analytics reports? How is testing typically tracked by Google Analytics?

    We would not expect LoadStorm test volume to show up in your Google Analytics. LoadStorm does not execute the javascript, and requests outside of your domain are blocked unless those other servers have also been verified. Servers like Google Analytics should be set to “Ignore”. So, the tracking by Analytics, ad servers, or other systems outside of your domain would not work. Some of these kinds of services have restrictions against load testing.

    Almost all of our customers don’t want our load testing tool traffic to show up in their marketing statistics (Google Analytics) because it essentially renders their analytics data worthless. Additionally, Google has limits on the amount of hits per period in their free GA account; otherwise you must upgrade to the paying account. There are also legal issues involved whereby Google doesn’t want automated tools like LoadStorm hammering their applications (even GA) – it is a violation of their standard terms and conditions. Thus, when a GA tracking javascript is found in a page that is involved in a LoadStorm test, that server is ignored automatically by our tool.

    Most of our clients use a different mechanism for watching the user activity of a load test. There are many tools for monitoring the website, including several excellent options for open source and commercial products such as New Relic, Loggly, and Hyperic. Most of these are very inexpensive or free, up to a certain level. These types of tools provide significant information regarding the user activity as well as important data on your architecture. For example, is your web server memory bound when response times start to increase drastically? Or did your database server hit 99% utilization when throughput drops?

    New Relic and Hyperic can monitor server activity from the local server using an agent. Loggly is a tool that has teamed up with New Relic for analyzing server logs which can make your life easier when trying to extrapolate the user activity from your server log.

    So to sum it up any javascript that reports traffic to third-party analytic tools will not work with LoadStorm. Instead useful tools for monitoring server usage and analyzing log files of user activity on the server under test (SUT) our encouraged and will work with LoadStorm.

  • Can your system run load tests behind my firewall for non-Internet applications?

    Unfortunately not. LoadStorm is only for web applications that can be reached from the Amazon cloud. If you have an intranet application, then opening a hole in your firewall and providing a URL or public IP address that can reach the intranet application will allow LoadStorm to run load tests against it.

     

  • How much does LoadStorm emulate the caching abilities of a real browser client? Will cache be reset across various sessions?

    The short answer is: If your browser caches it, LoadStorm will cache it for a virtual user. When a virtual user completes a script it closes the session. Then it clears all cookies and cache before repeating the script.

    Our load testing tool is designed to closely mimic the browser, but it offers three settings for caching behavior. There are a few exceptions. We do not cache HTML resources since these are often dynamically generated.

    Three settings for caching behavior
     

  • Does LoadStorm download images as a part of the load testing?

    Yes, and only if you want to. LoadStorm has a checkbox on the scenario edit page to turn download images on or off. There is a similar option for controlling the download and execution of Javascript files.

    Normally performance issues arise in the process of generating the dynamic page, and web servers can serve static images very quickly; however, we understand that many of our clients have websites with a large amount of media-related content, and load test results from those sites could be significantly affected by not downloading that content. Image download is especially relevant to performance if bottlenecks are in bandwidth or other network communications.

     

  • What’s the difference between “Concurrent Users”, “Sessions”, and “Total Users”?

    load testing browser emulationThe starting and peak number of virtual users in LoadStorm represent concurrent users or the number of simulated users at a particular point in time. Each concurrent user will last for the duration of the script. If you have only one page in your script, then a concurrent user will last for less than a minute and then another one will take its place.

    For example, if you have a test with one script having one page, starting at 10 concurrent users and peak of 10 concurrent users for 20 minutes, there will be 200-300 total users simulated (and same number of sessions). LoadStorm will create a new session for each virtual user, but it will not maintain the session after the script has ended, so your session should time out as you have configured it.

    The LoadStorm test parameters simulate “concurrent” users not total users. The total simulated users will generally be much higher than concurrent users, but the actual count depends on lots of other factors such as the number of pages in the script, think times, etc.

     

  • Why am I not getting as many page views as expected for the number of concurrent users in my test?

    Page views is a good term when reviewing Google Analytics data for traffic analysis. However, when reviewing load test data, page views can be a confusing metric. We do not track page views; rather we track requests and responses. Let’s look at the difference and how LoadStorm measures interactions with the server.

    The page views per virtual user should stay consistent as long as the response time is consistent and no errors. In some tests, the response time and errors are significant and increase over the duration of the test. For example, we see some load tests that have a large number of timeouts (no response to the request in 35 seconds). That means that LoadStorm cannot move to the next request for a virtual user until the response is received or we timeout the request. This drastically changes the equation you may be using to calculate the number of pageviews. Specifically, in one test there were 143,553 errors out of 1,345,534 total requests during a 30 minute test. This failure rate greatly decreased the number of requests LoadStorm was sending during the test.

    Also, if a request is sent for a step, LoadStorm is using the response header to capture the content type. A normal HTML page in a test returns, “text/html; charset=UTF-8″. When no response is received, LoadStorm assigns the response content type as, “NONE”. This affects the summary report table because the NONE responses are lumped into the OTHER category rather than the HTML category. Therefore, many of the 143k errors are shown in the OTHER category – not in the HTML category. That means that many of the HTML requests are not even shown in the table as HTML (or page views).

    The bottom line is that it’s mathematically invalid to expect linear correlation between page views, requests/responses, and vusers if the target server is not responding well. Everyone has a different definition of test success and failure. Some believe that greater than 1% error rate is a reason to stop the test. Others believe that an average response time greater than 3 seconds is a failure. Regardless how you define failure, timeouts and other server errors degrade the value of your load test metrics.

     

  • While I am running a load test and X number of users are hitting my web app, does this mean every second X number of requests are being sent to my application?

    The short answer is no. LoadStorm reporting graphs include a metric called Requests Per Second, and that is what you are describing.

    A virtual or simulated user does what a “real” user does as specified by the scripts and pages that you have created in LoadStorm. If there are X number of users, then there are X script instances running at that particular time.

    When creating a script, you can adjust the “Think Times” which is how much time there is after each page. The actual pause is randomly generated to give a more realistic load against the server. As an example, a script which makes a single request with an average think time of 30 seconds will have an average of 2 requests per minute per user. It gets more complicated when there are many GET requests like Javascript, CSS, and images. LoadStorm makes up to six GET requests at a time asynchronously so as soon as one of the first six completes another starts. This means that each response time has an impact on the number of requests per second.

    For a nice visual of how this works see the Waterfall tab within your script:

    Example of the Waterfall tab

     

  • What happens to a virtual user when something fails or times out in a scenario?

    load testing vuserThe vuser continues to move to the next step in its scenario. For instance, if a JavaScript file fails to download during step 3:

    • That error will be logged in LoadStorm for reporting
    • That request will be terminated
    • All other requests for step 3 will be attempted as usual
    • The vuser will begin step 4 as defined in the scenario

    If any particular resource times out within a step, some of the step processing may not finish. This depends on what kind of resource fails and at what point in the step. If an image fails, this is relatively harmless. The images following will not be requested, but everything else will continue normally. If the html fails with a timeout, then the step will stop and no images or other resources will be requested. Images, javascript and css all depend on the html, so none will be requested for that step unless the html is received without error.

    The step following a step with an error will work normally to the extent that it can. As with a browser, you cannot click on a link or submit a form if you do not currently have a valid page available with which to work. You can always type in a new url and go from there. Likewise, in LoadStorm if the next step is an open step, then previous steps will have no effect except for possibly login or cookie state. If the next step is a click or form, then the previous html must have been received for the step to function. If the step cannot work because the previous html timed out or otherwise failed, then that step will show no requests or errors and the next step will be attempted after the expected pause.

    There is currently only one situation where a scenario will restart before all of its steps have run. This is a step timeout where a step starts but does not finish before 5 minutes have passed. A step includes getting all resources within the step such as html, javascript, images, css, etc. So, it is not inconceivable that this timeout occurs if there are 50, 100 or more resources in a step, but it is long enough that it is pretty rare.

     

  • What happens to virtual users as test scenarios are running?

    LoadStorm creates the starting concurrent users within seconds of the test starting. For example, if you set the test to start at 1,000 users then you will have requests coming from those 1,000 users in the first 30 seconds. It then processes each virtual user through the script assigned to it. The first step, for example your home page, is requested first. The HTML is sent from your server and LoadStorm receives it. All other resources recorded (images, Javascript, CSS, and XML) are requested separately and sequentially.

    When all responses in a page are returned, the system pauses a random time between Min & Max Think Time pause (you can edit this), then it requests the next page. That continues until the virtual user reaches the end of the script, whereupon that VUser is “retired”. After about 30 seconds, another VUser is created and the process starts again. The cache for the new virtual user starts empty and does not remember cookies or static resources for the previous VUser.

     

  • Does LoadStorm’s request header allow for compression such as gzip?

    There is currently no functionality to add gzip to the LoadStorm request header. We hope to add that feature in an upcoming release. “Accept-Encoding: gzip, deflate” cannot be sent in the request generated by LoadStorm.

     


Analyzing Test Results

  • What does “response time” mean?

    load testing info
    LoadStorm calculates the response time in the following manner:

    The total time taken between the first byte of the request sent and the last byte of the response received for all requests made. This includes the network latency and the time for the server to respond.

    The primary page normally includes the HTML, CSS, and Javascript files. However, the primary page could be another resource such as a PDF, video, or Flash file. The response time may appear slightly faster than observed by you in a browser because it does not include time for a browser to render the page. Of course render time will be affected by browser type, operating system, and horsepower of the client computer.

    It is also important to note that LoadStorm’s servers probably have faster internet connections than many homes and offices.

     

  • How does it generate HTTP traffic against my web application?

    LoadStorm is a truly distributed application that leverages the power of Amazon Web Services to scale on demand with processing power and bandwidth as needed to test the largest web projects. As you crank up the testing load from 200 to 1,000,000 virtual users, LoadStorm is automatically adding machines for you (as many as necessary) from Amazon’s data centers to handle the processing. When your tests are done and the extra machines are not needed, they are turned off to wait for another test.

    A script is processed as a sequential series of pages that you have defined in the recording. Once a page finishes, the VUser pauses for the specified time (between Min & Max “Think Time” in the script), then the next page proceeds. After the last page completes, the VUser context (session, cache, cookies) is cleared and the script starts from the beginning again with the same VUser. This same VUser will use different data upon script repeat if a CSV of User Data or dynamic response text has been parameterized within the script. As the concurrent users scale up they each begin the script for the first time, and also repeat the script after each completion.

    You define the number of concurrent users you want at the start and peak of the test. You can also define a ramp down period to see how well your system recovers as the traffic subsides.

    Scripts are repeated until the test is done as defined by the test duration field (in minutes) from when you began the load test.

     

  • How do I stop a test that has killed my server?

    You can press the “Stop Execution” button during a load test. To do this:

    1. click ANALYZE in the left navigation
    2. double-click the test run that is currently in progress
    3. near the top-right in the filters area is the “Stop Execution” button

    How to stop a load test in progress
     

  • Why does the Summary Table of test results show my missing JPGs in the HTML row?

    If one of your web pages contains a reference to an image that is not available on your system, then a response header will be sent to LoadStorm with Status Code = 404. Additionally, your web server will send an HTML page indicating the 404 “Not Found” page to alert the user. That’s standard practice in web applications.

    The reason LoadStorm reports the 404 errors as HTML is because your web server is sending our system the Content Type = “HTML” attribute. LoadStorm separates the data by the Content Type.

     

  • Why aren’t the 500 users evenly distributed across all 5 of my servers behind a round robin load balancer?

    A round robin load balancer will not work very well for testing your configuration unless you are using a much higher number of virtual users (at least 1000 or more). Each of our simulation servers will resolve the DNS name independently of the others, but each of our servers will simulate hundreds of users. I would expect a peak load of 500 users to get routed to more than one of your servers, but probably not all 5 of them. A peak load of 200 users or less is likely to hit only one of your servers.

     

  • Will the load test show up in the Google Analytics reports? How is testing typically tracked by Google Analytics?

    We would not expect LoadStorm test volume to show up in your Google Analytics. LoadStorm does not execute the javascript, and requests outside of your domain are blocked unless those other servers have also been verified. Servers like Google Analytics should be set to “Ignore”. So, the tracking by Analytics, ad servers, or other systems outside of your domain would not work. Some of these kinds of services have restrictions against load testing.

    Almost all of our customers don’t want our load testing tool traffic to show up in their marketing statistics (Google Analytics) because it essentially renders their analytics data worthless. Additionally, Google has limits on the amount of hits per period in their free GA account; otherwise you must upgrade to the paying account. There are also legal issues involved whereby Google doesn’t want automated tools like LoadStorm hammering their applications (even GA) – it is a violation of their standard terms and conditions. Thus, when a GA tracking javascript is found in a page that is involved in a LoadStorm test, that server is ignored automatically by our tool.

    Most of our clients use a different mechanism for watching the user activity of a load test. There are many tools for monitoring the website, including several excellent options for open source and commercial products such as New Relic, Loggly, and Hyperic. Most of these are very inexpensive or free, up to a certain level. These types of tools provide significant information regarding the user activity as well as important data on your architecture. For example, is your web server memory bound when response times start to increase drastically? Or did your database server hit 99% utilization when throughput drops?

    New Relic and Hyperic can monitor server activity from the local server using an agent. Loggly is a tool that has teamed up with New Relic for analyzing server logs which can make your life easier when trying to extrapolate the user activity from your server log.

    So to sum it up any javascript that reports traffic to third-party analytic tools will not work with LoadStorm. Instead useful tools for monitoring server usage and analyzing log files of user activity on the server under test (SUT) our encouraged and will work with LoadStorm.

  • How much does LoadStorm emulate the caching abilities of a real browser client? Will cache be reset across various sessions?

    The short answer is: If your browser caches it, LoadStorm will cache it for a virtual user. When a virtual user completes a script it closes the session. Then it clears all cookies and cache before repeating the script.

    Our load testing tool is designed to closely mimic the browser, but it offers three settings for caching behavior. There are a few exceptions. We do not cache HTML resources since these are often dynamically generated.

    Three settings for caching behavior
     

  • What’s the difference between “Concurrent Users”, “Sessions”, and “Total Users”?

    load testing browser emulationThe starting and peak number of virtual users in LoadStorm represent concurrent users or the number of simulated users at a particular point in time. Each concurrent user will last for the duration of the script. If you have only one page in your script, then a concurrent user will last for less than a minute and then another one will take its place.

    For example, if you have a test with one script having one page, starting at 10 concurrent users and peak of 10 concurrent users for 20 minutes, there will be 200-300 total users simulated (and same number of sessions). LoadStorm will create a new session for each virtual user, but it will not maintain the session after the script has ended, so your session should time out as you have configured it.

    The LoadStorm test parameters simulate “concurrent” users not total users. The total simulated users will generally be much higher than concurrent users, but the actual count depends on lots of other factors such as the number of pages in the script, think times, etc.

     

  • While I am running a load test and X number of users are hitting my web app, does this mean every second X number of requests are being sent to my application?

    The short answer is no. LoadStorm reporting graphs include a metric called Requests Per Second, and that is what you are describing.

    A virtual or simulated user does what a “real” user does as specified by the scripts and pages that you have created in LoadStorm. If there are X number of users, then there are X script instances running at that particular time.

    When creating a script, you can adjust the “Think Times” which is how much time there is after each page. The actual pause is randomly generated to give a more realistic load against the server. As an example, a script which makes a single request with an average think time of 30 seconds will have an average of 2 requests per minute per user. It gets more complicated when there are many GET requests like Javascript, CSS, and images. LoadStorm makes up to six GET requests at a time asynchronously so as soon as one of the first six completes another starts. This means that each response time has an impact on the number of requests per second.

    For a nice visual of how this works see the Waterfall tab within your script:

    Example of the Waterfall tab

     


Building Test Scripts

  • How do I build a script and schedule a test run?

    Excellent question. Click here for step-by-step instructions to begin using LoadStorm.

     

  • Why do I need to verify my server? How do I do that?

    This is a very common question. A example email may look something like this:

    I am unable to schedule a test run because LoadStorm shows that I need to verify a bunch of servers:

    which-servers-need-to-be-verified

    Click to zoom.

    I am not sure what to do now…Why do I have to verify these servers?

    Sincerely, Mr. WPL

    Are You Load Testing Facebook?

    From the selected script LoadStorm found that this test plan will be hitting servers that aren’t verified (authorized for testing). Put another way, these scripts contain requests that go to servers that do not belong to Mr. WPL in the example email, and LoadStorm gets in trouble when our customers load test other people’s servers without their permission.

    Does Mr. WPL really want to hit Facebook and Pinterest as part of his load test? If so, do those 3rd party servers contribute a crucial part of his server’s ability to handle load? If so, does he have their permission to hammer on their servers? Probably not. They tend to get upset when someone utilizes a cloud load testing tool to simulate thousands of VUsers against their social media platform. Please read the terms and conditions of use for those services.

    And where do those VUsers originate? LoadStorm. So who gets the phone call from their lawyers? We do.

    Prevent Denial of Service Attacks

    Our verification process is designed to prevent LoadStorm from being used as a Denial of Service (DOS) weapon. By placing a code on your home page that LoadStorm can request and confirm, our system confirms that you have the control over that server; thus, it is safe for our tool to hammer it with VUsers.

    Recommendation

    To verify a server follow the instructions in our learning center or watch this video tutorial.

    Ignore all third-party servers that are not crucial to your web application. LoadStorm will not make any requests to ignored servers. Remember the goal is to test the scalability of your web servers, and not those of third-parties. If you feel there is a third-party server that is crucial to your testing needs please email them and copy support@loadstorm.com requesting permission from their company to let you use our software to load test their web servers and yours as part of your testing needs.

  • How should I set up scripts to simulate real traffic?

    Most people create several scripts to accurately reflect different types of users. Scripts execute simultaneously in order to hit your site with the correct mix of traffic. You control the weighting of each script by giving it a number relative to the others. For example, if you have 2 scripts and you give one a weighting of 4 and the other a weighting of 1, then the first will generate 80% of the traffic and the second 20%. This 80/20 could be realistic for a retail e-commerce site where most of the people (80%) are browsing the product catalog, while a minority of people (20%) are going through a buying experience. You can of course throw in a low-weight scenario to represent some internal employees adding products or issuing refunds.

    Another way of entering the weight of the traffic is to just enter the exact percentage you would like each script to carry. For the example above you would simply enter 80 and 20. The only thing you have to be very careful about is that the percentages add up to 100. Otherwise, it will work like the above description, which is different.

    There is no limit to the number of scripts you can create, and you can use many scripts in a test. We encourage you to build as many as necessary to accurately reflect the real traffic patterns on your site. That will give you the most meaningful metrics for tuning your applications.

    We also simulate the user “think time” with random pauses after each page. Real users click on something, then read the page or fill in a form, and then click on something else. Thus, you don’t want to have a simulated virtual user clicking on something every .5 seconds – it would not give you meaningful results if you want to see how your site holds up in the real world.

    Also see: Virtual User Calculations about calculating the amount of virtual users needed for a realistic test.

  • Why should I ramp up my test load volume over time?

    It helps with performance engineering. We recommend to not begin a large test at the maximum number of virtual users. Rather than run a test for 10 minutes beginning with 5,000 and ending with 5,000, we suggest to start smaller, for example, with 500 users and increase to 5,000 over an hour. This approach has the advantage of allowing your system to get all the parts working properly at smaller load (e.g. caches, threads, database connections) before the heavier volume starts exposing potential bottlenecks in your application. Also if the target server begins to fail during the test this provides more granularity into the number of concurrent users where failure begins. If the test starts at 5,000 and fails all that can be learned from the results is that it fails somewhere between 1 and 5,000 which isn’t very useful.

    From what we’ve seen in thousands of load tests with LoadStorm, it is common to see useful patterns in the metrics before the peak traffic is reached, and these metrics point to areas of performance limitations that can be tuned.

    Load testing is invariably coupled with performance tuning – an iterative process of test/tune/test/tune. Therefore, we recommend you ramp the volume in order to get more value from the test results.

     

  • Other than IP address, is there any way to identify LoadStorm requests from normal traffic?

    Yes, there are two additional ways for you to white list our traffic:

    1. You can customize the user agent for each script which your firewall can see in the request headers of inbound requests. However, if your test scripts contains HTTPS requests, then those headers will be encrypted. Visit our learning center for more information on modifying a script’s default settings such as the User Agent.
    2. If you are not testing against your production server, we recommend trying a non-standard port number to identify the traffic as load testing. LoadStorm allows you to use any port number and can be designated for each target server.

    Thumbs Up Button

  • Can I create a test script where virtual users login to my web application? If so, how do I get my test user file into LoadStorm?

    Yes. Once you’ve created a script, you may modify form POSTs in order to log into your application with test user data.

    We recommend you visit our learning center, where we go over all of the options for parameterization in a script. A brief explanation is given below, but we also offer video tutorials on common examples such as parameterizing a login form.

    While creating a recording using developer tools in your browser, log into your application, and save the recording as a HAR file. Then upload the recording into your LoadStorm account. Open your script and select the parameterization tab.

    After that, click BUILD in the left navigation and switch to the User Data tab at the top. Here you can upload your CSV which contains necessary login info (username/email, passwords). Note that the first row’s data will be read as column headers and not row data. You can also choose to “generate data” on-the-fly in LoadStorm. If you need to embed a comma inside a particular data field, simply place quotation marks around all field data in the file.

    Finally, edit the necessary script and find the POST request that submits the form. Modify the form to include Custom Data with CSV user information. To add the test user data, you must select “custom” under the form modifications options, and select the data from a CSV file.

    This simulates traffic more realistically. Instead of simulating one user logging in over and over, many different users log in instead.

  • How Can I Increase the Requests per Second in My Test?

    Here are some thoughts on increasing RPS in LoadStorm tests:

    1. Reducing the think time added after each page in a script; the minimum pause is 5 seconds, so set both min & max to 5 seconds.
    2. Increasing the number of concurrent users.
    3. Reducing the response time from your server which decreases the time LoadStorm has to wait for a request to complete.
    4. Compressing the size of the response is one way to decrease the time it takes for the request to complete as well as save on throughput.

    RPS will always be impacted by the types of pages/resources your scripts are requesting. The way it works is that LoadStorm cranks up the starting concurrent users shortly after test starts. It then walks each user through the script assigned to it. The GET requests will be made in asynchronous bursts of up to 6 at a time such as HTML, images, JS, CSS, and XML. Each POST request will be made on its own to ensure that any dynamic response text from earlier requests was available for to be passed into the POST data. This also ensures that any response data from the POST is available for subsequent requests. When those responses are all returned, then the system pauses a random time between the Min & Max “Think Time”, then it requests the next page. This process continues until the end of the script. So, having more users going at the same time, without waiting more than a few hundred ms to get a response, will increase RPS.

     

  • Can I change the host file for my test? I need to hit a server that does NOT currently have a valid DNS name because we don’t want it found by customers until we launch the site.

    Unfortunately, our load testing tool doesn’t have a feature for what you need. Normally this is a simple process of putting a domain name to IP address mapping in a configuration file on a particular machine. The process is not as easy in our situation because we use dynamically instantiated servers from the Amazon cloud; thus, we do not know what IP addresses we will be using before you run your tests. We would need to create new images specifically for your tests, and we would need to make changes to our core launching software that instantiates the images so that new machines are launched with this particular host mapping. If you would like us to coordinate this custom process, please contact our support. Additional charges per test will apply.

     


Getting Started

  • How do I build a script and schedule a test run?

    Excellent question. Click here for step-by-step instructions to begin using LoadStorm.

     

  • What does LoadStorm do?

    load testing info
    LoadStorm™ is a web-based load testing tool for simulating what users do with a web site or web application. You use it to build tests that send requests to your server in the same way that a user’s browser sends requests to your server. But these tests are executed by our automated systems rather than by a user, so they can be done repeatedly and in large numbers simultaneously. They can also be built using our tool in such a way as to simulate a large number of different users with different tasks to perform.

     

  • How much does it cost?

    load testing cost
    It depends on how many virtual users you need for testing, and it depends on if you would like to use our consulting services.

    Creating an account is free, but a free account is limited to 50 virtual users to try out our product. By default, we offer four pricing plans. Each plan scales up in cost, but offers more benefits such as bundled consulting hours.

    For more details, please see Load Testing Cost.

     

  • Why do I need to verify my server? How do I do that?

    This is a very common question. A example email may look something like this:

    I am unable to schedule a test run because LoadStorm shows that I need to verify a bunch of servers:

    which-servers-need-to-be-verified

    Click to zoom.

    I am not sure what to do now…Why do I have to verify these servers?

    Sincerely, Mr. WPL

    Are You Load Testing Facebook?

    From the selected script LoadStorm found that this test plan will be hitting servers that aren’t verified (authorized for testing). Put another way, these scripts contain requests that go to servers that do not belong to Mr. WPL in the example email, and LoadStorm gets in trouble when our customers load test other people’s servers without their permission.

    Does Mr. WPL really want to hit Facebook and Pinterest as part of his load test? If so, do those 3rd party servers contribute a crucial part of his server’s ability to handle load? If so, does he have their permission to hammer on their servers? Probably not. They tend to get upset when someone utilizes a cloud load testing tool to simulate thousands of VUsers against their social media platform. Please read the terms and conditions of use for those services.

    And where do those VUsers originate? LoadStorm. So who gets the phone call from their lawyers? We do.

    Prevent Denial of Service Attacks

    Our verification process is designed to prevent LoadStorm from being used as a Denial of Service (DOS) weapon. By placing a code on your home page that LoadStorm can request and confirm, our system confirms that you have the control over that server; thus, it is safe for our tool to hammer it with VUsers.

    Recommendation

    To verify a server follow the instructions in our learning center or watch this video tutorial.

    Ignore all third-party servers that are not crucial to your web application. LoadStorm will not make any requests to ignored servers. Remember the goal is to test the scalability of your web servers, and not those of third-parties. If you feel there is a third-party server that is crucial to your testing needs please email them and copy support@loadstorm.com requesting permission from their company to let you use our software to load test their web servers and yours as part of your testing needs.

  • How should I set up scripts to simulate real traffic?

    Most people create several scripts to accurately reflect different types of users. Scripts execute simultaneously in order to hit your site with the correct mix of traffic. You control the weighting of each script by giving it a number relative to the others. For example, if you have 2 scripts and you give one a weighting of 4 and the other a weighting of 1, then the first will generate 80% of the traffic and the second 20%. This 80/20 could be realistic for a retail e-commerce site where most of the people (80%) are browsing the product catalog, while a minority of people (20%) are going through a buying experience. You can of course throw in a low-weight scenario to represent some internal employees adding products or issuing refunds.

    Another way of entering the weight of the traffic is to just enter the exact percentage you would like each script to carry. For the example above you would simply enter 80 and 20. The only thing you have to be very careful about is that the percentages add up to 100. Otherwise, it will work like the above description, which is different.

    There is no limit to the number of scripts you can create, and you can use many scripts in a test. We encourage you to build as many as necessary to accurately reflect the real traffic patterns on your site. That will give you the most meaningful metrics for tuning your applications.

    We also simulate the user “think time” with random pauses after each page. Real users click on something, then read the page or fill in a form, and then click on something else. Thus, you don’t want to have a simulated virtual user clicking on something every .5 seconds – it would not give you meaningful results if you want to see how your site holds up in the real world.

    Also see: Virtual User Calculations about calculating the amount of virtual users needed for a realistic test.

  • Why should I ramp up my test load volume over time?

    It helps with performance engineering. We recommend to not begin a large test at the maximum number of virtual users. Rather than run a test for 10 minutes beginning with 5,000 and ending with 5,000, we suggest to start smaller, for example, with 500 users and increase to 5,000 over an hour. This approach has the advantage of allowing your system to get all the parts working properly at smaller load (e.g. caches, threads, database connections) before the heavier volume starts exposing potential bottlenecks in your application. Also if the target server begins to fail during the test this provides more granularity into the number of concurrent users where failure begins. If the test starts at 5,000 and fails all that can be learned from the results is that it fails somewhere between 1 and 5,000 which isn’t very useful.

    From what we’ve seen in thousands of load tests with LoadStorm, it is common to see useful patterns in the metrics before the peak traffic is reached, and these metrics point to areas of performance limitations that can be tuned.

    Load testing is invariably coupled with performance tuning – an iterative process of test/tune/test/tune. Therefore, we recommend you ramp the volume in order to get more value from the test results.

     

  • Why doesn’t my test start immediately?

    LoadStorm needs a little time to prepare the test resources such as the load generation servers. Our system is using each selected script(s) to pre-calculate the virtual users’ actions, how many servers will need to be instantiated from the cloud, random think times, etc. This produces essentially a test roadmap that our Scheduler and Summarizer modules use to coordinate all the HTTP traffic and correctly capture the metrics. Depending on the number of concurrent users and the duration of your test, the preparation of a load test can take 2 to 15 minutes.

     

  • What are the IP address ranges used by LoadStorm? We need to white list them.

    Because we use the Amazon EC2 cloud to dynamically instantiate load generation servers we don’t have a specific static IP address. However, we can provide the possible ranges for each EC2 data center or geographic location that we use. The only groups of ranges that we do not use are GovCloud, and Beijing. The list of ranges are no longer updated from Amazon’s blog post about EC2 public IP ranges.

    Amazon now maintains the list of their EC2 public IP ranges in this JSON file:
    https://ip-ranges.amazonaws.com/ip-ranges.json

  • If I stop a test before reaching the peak users, are the rest of my users wasted?

    You can terminate a test early because the target server fails. When this happens in a test using virtual users, those users may seem “wasted”.

    Customers have asked us why the system deducted 5,000 VUsers from their account when the test only lasted 10 minutes and stopped at 500 users. The reason LoadStorm does not prorate the VUsers to the minute or to the actual peak users is because we incur the full costs of the 5,000 VUser test.

    Amazon’s cloud is elastic, and we “rent” the servers needed for your tests. However, there is a minimum of 1 hour increments that Amazon will bill us. For example, if your test needs 20 servers to hit your defined peak users, we buy those 20 servers upfront before your test actually starts. Then if your load test dies 5 minutes into it, we are still charged as if the test used all 20 servers for 1 hour.

    Thus, we recommend that you start with smaller tests to verify that you have all of the environmental factors properly configured. Growing the volume in several tests that increase in volume will potentially eliminate the “wasted” feeling of throwing 5,000 users at a server that fails at 500 users.

     

  • Why not let our users test the system’s capacity?

    For some projects, this is an acceptable approach. For some internal applications, apps that are very simple will realistically expect light traffic. The best plan may be to get it done as quickly as possible. You have to ask yourself what is the risk of the application not working at some point. For many businesses, getting a large influx of new users is the peak of success, and that success would be shattered by having the new users unable to use the system, unable to buy your product and unhappy with your company.

     


Only LoadStorm LITE

  • When configuring a scenario step that clicks a link, how does LoadStorm follow the link?

    LoadStorm follows links when the session ID is appended to the URL query string because it searches for the text between the open and close anchor tags. If it doesn’t find that text, then the fall back is to match the XPATH. We stored the XPATH when the scenario was created (essentially an address inside the page’s DOM). The XPATH is a unique identifier that allows us to click on the anchor such as an image or the href attribute or execute the Javascript.

     

  • What is the “Average Duration” for a scenario?

    Average Duration is the time taken for ALL steps to complete (see step duration calculation), plus the average pause times between steps. Thus, it is the mean time for a specific vuser’s “life”. The actual duration fluctuates with the amount of pause assigned to that vuser. The Average Duration is shown above the steps on the Scenario page.

     

  • How is “Duration” for a scenario step calculated?

    Duration is the time taken for a scenario step to complete, which is the sum of time to last byte for each request in the step (including HTML, JS, CSS, images, etc.). The Average Duration shown above the steps on the Scenario page is the total of all step durations plus the average pause times between steps.

     

  • What is the “Form data set” option when creating a new scenario? Why are there options for Tiny, Small, and Huge?

    The Form Data Set field that you see on the Scenario settings page is a mechanism to select an input source for form submission. Using the Form Data Sets functionality, it is possible to submit forms such as a login form and fill-in the fields (e.g. username, password) with data from your system. Other common types of form submission are search terms, credit card numbers, and profile preferences. Any type of form field that can be filled in with text can utilize the Form Data Sets.

    You may upload your own data to be used for form submission, or you may use one of our built-in form data sets.

    The Tiny, Small, and Huge form data sets are built-into LoadStorm. The only real difference between the three files are the number of rows. Tiny only has 10 records. Small has 100, and Huge has 10,000 records of user information. They contain user information:

    • First Name
    • Last Name
    • Username
    • Password
    • Email address

    Many of our customers use one of our built-in form data sets for registration on their web application or for logging in virtual users.

    If you are going to upload your own data into LoadStorm, here is an example of the process for getting user credentials from your system to LoadStorm:

    1. Export a CSV file of user information from your user table(s).
    2. The first row of the CSV must contain the names of the columns, and these column names will be mapped to the field name in the step that submits the form.
    3. The CSV file is uploaded by clicking on the UPLOAD DATA button on the Build tab. This button will be near the bottom, under the Form Data Sets section of the page.
    4. You must give the file a label so that you can choose that file to be used at the scenario level.

    Once you have the CSV data in LoadStorm (you should see it on the Build tab under the heading Form Data Sets), then you need to edit the scenario and choose the appropriate data set based on the label you assigned when uploading it. This essentially connects the CSV file to your scenario. Then when you add a step for form submission, you will be able to map the fields in the form to the columns of data in the Form Data Set.

    The LoadStorm Tour has a video tutorial showing how to create a login form submission. Watch the video on YouTube

     

  • Do you provide a URL import functionality? I need to perform a test that includes roughly 100,000 URLs, however I do not want to have to enter in each URL individually.

    The concept is to upload a file of strings that can be substituted in a single step of a scenario. Most customers use this functionality to append a query string on the end of a domain. You can produce a CSV with one or more columns containing the strings. Then you add a step that opens a page, and there is specific syntax for mapping the string from a column of the CSV into the URL requested by the step.

    Here’s an example of what would be used for the step:

    http://cia-factbook.s3.amazonaws.com#{FileOfPaths.pathFull1}

    The hash and open brace are significant because they tell LoadStorm to substitute the text. “FileOfPaths” is the label for the CSV I uploaded into my LoadStorm account. “pathFull1″ is the column name inside FileOfPaths that contains the string I want to append on the domain.

    To upload the CSV (form data set) of your URLs/query strings into LoadStorm:

    1. Export a CSV file with one or more columns containing the URLs.
    2. The first row of the CSV must contain the names of the columns, and these column names will be mapped to the substitution point in the step indicated by the #{ } delimiters.
    3. The CSV file is uploaded by clicking on the UPLOAD DATA button on the Build tab. This button will be near the bottom, under the Form Data Sets section of the page.
    4. You must give the file a label so that you can choose that file to be used at the scenario level.

    Once you have the CSV data in LoadStorm (you should see it on the Build tab under the heading Form Data Sets), then you need to edit the scenario and choose the appropriate data set based on the label you assigned when uploading it. This essentially connects the CSV file to your scenario. Then when you add a step for using the URL/query string, you will be able to place #{formDataSet.columnName} from the Form Data Set.

     

  • Why does my step not show my images or contents aren’t displaying properly?

    When adding a step in LoadStorm, our emulator makes the request to your web application and receives the response containing the HTML code for the page. Our tool then parses the HTML to determine what additional resources (e.g. images, Javascript libraries) need to be requested for that page.

    Each of those resources are requested and received. LoadStorm saves all of the HTML and other resources in our database along with other information it needs to reproduce the sequence of events for this step during a load test. The visual rendering you see at the bottom of the step is an iFrame where our tool places a copy of what was received during the step creation process. It is NOT an accurate rendering as you would see it in a full browser without all of the other LoadStorm code and process “wrapped around” it. Essentially, what you are seeing is a page inside a page. The images may not display because of how they are referenced (e.g. absolute or relative) in the HTML code.

    The purpose of this iFrame display is to give the person building test scenarios confidence that the correct requests/responses were transacted for this step. The images you don’t see are still going to be requested during the load test, and all of the performance metrics such as response times will be accurate. Virtual users do not render the pages. No one will actually “see” the transactions during a load test because there is no need to see the pages, and our tool is measuring the Throughput, Error Rate, Response Times, Requests per Second, and Concurrent Users from many different load generation machines that process each virtual user going through the steps.

     

  • With a monthly subscription, are you able to add additional one-time VUsers?

    Yes, you may add Storm on Demand users for any load test. This allows you to spike test traffic to a much higher level than your subscription.

    For example, if you have a Hurricane subscription for 5,000 maximum concurrent vusers, you can purchase an additional 10,000 Storm on Demand users for a total of 15,000 vusers on a single test. Those 10,000 on demand users would be available for only one load test, and the 5,000 are available for other tests throughout the month.

     

  • Does LoadStorm download images as a part of the load testing?

    Yes, and only if you want to. LoadStorm has a checkbox on the scenario edit page to turn download images on or off. There is a similar option for controlling the download and execution of Javascript files.

    Normally performance issues arise in the process of generating the dynamic page, and web servers can serve static images very quickly; however, we understand that many of our clients have websites with a large amount of media-related content, and load test results from those sites could be significantly affected by not downloading that content. Image download is especially relevant to performance if bottlenecks are in bandwidth or other network communications.

     

  • Why am I not getting as many page views as expected for the number of concurrent users in my test?

    Page views is a good term when reviewing Google Analytics data for traffic analysis. However, when reviewing load test data, page views can be a confusing metric. We do not track page views; rather we track requests and responses. Let’s look at the difference and how LoadStorm measures interactions with the server.

    The page views per virtual user should stay consistent as long as the response time is consistent and no errors. In some tests, the response time and errors are significant and increase over the duration of the test. For example, we see some load tests that have a large number of timeouts (no response to the request in 35 seconds). That means that LoadStorm cannot move to the next request for a virtual user until the response is received or we timeout the request. This drastically changes the equation you may be using to calculate the number of pageviews. Specifically, in one test there were 143,553 errors out of 1,345,534 total requests during a 30 minute test. This failure rate greatly decreased the number of requests LoadStorm was sending during the test.

    Also, if a request is sent for a step, LoadStorm is using the response header to capture the content type. A normal HTML page in a test returns, “text/html; charset=UTF-8″. When no response is received, LoadStorm assigns the response content type as, “NONE”. This affects the summary report table because the NONE responses are lumped into the OTHER category rather than the HTML category. Therefore, many of the 143k errors are shown in the OTHER category – not in the HTML category. That means that many of the HTML requests are not even shown in the table as HTML (or page views).

    The bottom line is that it’s mathematically invalid to expect linear correlation between page views, requests/responses, and vusers if the target server is not responding well. Everyone has a different definition of test success and failure. Some believe that greater than 1% error rate is a reason to stop the test. Others believe that an average response time greater than 3 seconds is a failure. Regardless how you define failure, timeouts and other server errors degrade the value of your load test metrics.

     

  • What happens to a virtual user when something fails or times out in a scenario?

    load testing vuserThe vuser continues to move to the next step in its scenario. For instance, if a JavaScript file fails to download during step 3:

    • That error will be logged in LoadStorm for reporting
    • That request will be terminated
    • All other requests for step 3 will be attempted as usual
    • The vuser will begin step 4 as defined in the scenario

    If any particular resource times out within a step, some of the step processing may not finish. This depends on what kind of resource fails and at what point in the step. If an image fails, this is relatively harmless. The images following will not be requested, but everything else will continue normally. If the html fails with a timeout, then the step will stop and no images or other resources will be requested. Images, javascript and css all depend on the html, so none will be requested for that step unless the html is received without error.

    The step following a step with an error will work normally to the extent that it can. As with a browser, you cannot click on a link or submit a form if you do not currently have a valid page available with which to work. You can always type in a new url and go from there. Likewise, in LoadStorm if the next step is an open step, then previous steps will have no effect except for possibly login or cookie state. If the next step is a click or form, then the previous html must have been received for the step to function. If the step cannot work because the previous html timed out or otherwise failed, then that step will show no requests or errors and the next step will be attempted after the expected pause.

    There is currently only one situation where a scenario will restart before all of its steps have run. This is a step timeout where a step starts but does not finish before 5 minutes have passed. A step includes getting all resources within the step such as html, javascript, images, css, etc. So, it is not inconceivable that this timeout occurs if there are 50, 100 or more resources in a step, but it is long enough that it is pretty rare.

     

  • What happens to virtual users as test scenarios are running?

    LoadStorm creates the starting concurrent users within seconds of the test starting. For example, if you set the test to start at 1,000 users then you will have requests coming from those 1,000 users in the first 30 seconds. It then processes each virtual user through the script assigned to it. The first step, for example your home page, is requested first. The HTML is sent from your server and LoadStorm receives it. All other resources recorded (images, Javascript, CSS, and XML) are requested separately and sequentially.

    When all responses in a page are returned, the system pauses a random time between Min & Max Think Time pause (you can edit this), then it requests the next page. That continues until the virtual user reaches the end of the script, whereupon that VUser is “retired”. After about 30 seconds, another VUser is created and the process starts again. The cache for the new virtual user starts empty and does not remember cookies or static resources for the previous VUser.

     

  • Does LoadStorm’s request header allow for compression such as gzip?

    There is currently no functionality to add gzip to the LoadStorm request header. We hope to add that feature in an upcoming release. “Accept-Encoding: gzip, deflate” cannot be sent in the request generated by LoadStorm.

     


Pricing

  • How much does it cost?

    load testing cost
    It depends on how many virtual users you need for testing, and it depends on if you would like to use our consulting services.

    Creating an account is free, but a free account is limited to 50 virtual users to try out our product. By default, we offer four pricing plans. Each plan scales up in cost, but offers more benefits such as bundled consulting hours.

    For more details, please see Load Testing Cost.

     

  • Does LoadStorm use a full browser for accessing the websites to test or does it emulate a browser?

    load testing ramp time
    LoadStorm PRO does not use a browser emulator, nor is it a full browser. Instead, it takes a HAR recording and builds a script from the recorded network requests to mimic a real browser while still allowing for a competitively costed solution. LoadStorm PRO also supports AJAX requests as well as REST and SOAP API calls. If we used an actual browser, we would trade-off the cost effectiveness of our solution for the ability to actually process javascript and other interactions exactly as a browser. Some solutions use actual browsers and cost you a great deal more money.

    We understand some web developers need real browsers and are willing to pay for it. Our load testing tool is designed to help those developers that are willing to achieve their load testing goals without a full browser and at a much lower cost.

     

  • If I stop a test before reaching the peak users, are the rest of my users wasted?

    You can terminate a test early because the target server fails. When this happens in a test using virtual users, those users may seem “wasted”.

    Customers have asked us why the system deducted 5,000 VUsers from their account when the test only lasted 10 minutes and stopped at 500 users. The reason LoadStorm does not prorate the VUsers to the minute or to the actual peak users is because we incur the full costs of the 5,000 VUser test.

    Amazon’s cloud is elastic, and we “rent” the servers needed for your tests. However, there is a minimum of 1 hour increments that Amazon will bill us. For example, if your test needs 20 servers to hit your defined peak users, we buy those 20 servers upfront before your test actually starts. Then if your load test dies 5 minutes into it, we are still charged as if the test used all 20 servers for 1 hour.

    Thus, we recommend that you start with smaller tests to verify that you have all of the environmental factors properly configured. Growing the volume in several tests that increase in volume will potentially eliminate the “wasted” feeling of throwing 5,000 users at a server that fails at 500 users.

     


Running Load Tests

  • How do I build a script and schedule a test run?

    Excellent question. Click here for step-by-step instructions to begin using LoadStorm.

     

  • I can’t run a load test because a server isn’t verified or ignored. Can you help me?

    Manage Servers

    Reach out to support@loadstorm.com for help with the verification process if you are stuck. For instructions on how to verify a server, please see our learning center‎ or watch this video tutorial.

    How does verification work?

    Our verification process works like Google Analytics. The idea is to prove it is your server so LoadStorm isn’t used for a denial-of-service attack. You can put an empty file that is named with a unique string generated by LoadStorm (i.e. loadstorm-<unique string>) into your root directory (doesn’t matter what is in it). So let’s say your target server is http://www.xyzcorp.com, then there should be a valid response if a request comes to your server for http://www.xyzcorp.com/loadstorm-90630fc588.html.

    Alternatively, if you don’t want to use the file you can embed the code in your default page using HTML comment tags. To continue the example, the HTML page received from requesting http://www.xyzcorp.com should contain a string within comment tags somewhere inside the document that matches “loadstorm-90630fc588″. LoadStorm will parse the page returned by your server exactly like a browser will. In that parsing, the verification string must appear somewhere for us to consider it verified.

    Example Code with a string in a comment tag:
    <html>
    <!– loadstorm-90630fc588 –>
    </html>

  • Why should I ramp up my test load volume over time?

    It helps with performance engineering. We recommend to not begin a large test at the maximum number of virtual users. Rather than run a test for 10 minutes beginning with 5,000 and ending with 5,000, we suggest to start smaller, for example, with 500 users and increase to 5,000 over an hour. This approach has the advantage of allowing your system to get all the parts working properly at smaller load (e.g. caches, threads, database connections) before the heavier volume starts exposing potential bottlenecks in your application. Also if the target server begins to fail during the test this provides more granularity into the number of concurrent users where failure begins. If the test starts at 5,000 and fails all that can be learned from the results is that it fails somewhere between 1 and 5,000 which isn’t very useful.

    From what we’ve seen in thousands of load tests with LoadStorm, it is common to see useful patterns in the metrics before the peak traffic is reached, and these metrics point to areas of performance limitations that can be tuned.

    Load testing is invariably coupled with performance tuning – an iterative process of test/tune/test/tune. Therefore, we recommend you ramp the volume in order to get more value from the test results.

     

  • Why doesn’t my test start immediately?

    LoadStorm needs a little time to prepare the test resources such as the load generation servers. Our system is using each selected script(s) to pre-calculate the virtual users’ actions, how many servers will need to be instantiated from the cloud, random think times, etc. This produces essentially a test roadmap that our Scheduler and Summarizer modules use to coordinate all the HTTP traffic and correctly capture the metrics. Depending on the number of concurrent users and the duration of your test, the preparation of a load test can take 2 to 15 minutes.

     

  • How does it generate HTTP traffic against my web application?

    LoadStorm is a truly distributed application that leverages the power of Amazon Web Services to scale on demand with processing power and bandwidth as needed to test the largest web projects. As you crank up the testing load from 200 to 1,000,000 virtual users, LoadStorm is automatically adding machines for you (as many as necessary) from Amazon’s data centers to handle the processing. When your tests are done and the extra machines are not needed, they are turned off to wait for another test.

    A script is processed as a sequential series of pages that you have defined in the recording. Once a page finishes, the VUser pauses for the specified time (between Min & Max “Think Time” in the script), then the next page proceeds. After the last page completes, the VUser context (session, cache, cookies) is cleared and the script starts from the beginning again with the same VUser. This same VUser will use different data upon script repeat if a CSV of User Data or dynamic response text has been parameterized within the script. As the concurrent users scale up they each begin the script for the first time, and also repeat the script after each completion.

    You define the number of concurrent users you want at the start and peak of the test. You can also define a ramp down period to see how well your system recovers as the traffic subsides.

    Scripts are repeated until the test is done as defined by the test duration field (in minutes) from when you began the load test.

     

  • How Can I Increase the Requests per Second in My Test?

    Here are some thoughts on increasing RPS in LoadStorm tests:

    1. Reducing the think time added after each page in a script; the minimum pause is 5 seconds, so set both min & max to 5 seconds.
    2. Increasing the number of concurrent users.
    3. Reducing the response time from your server which decreases the time LoadStorm has to wait for a request to complete.
    4. Compressing the size of the response is one way to decrease the time it takes for the request to complete as well as save on throughput.

    RPS will always be impacted by the types of pages/resources your scripts are requesting. The way it works is that LoadStorm cranks up the starting concurrent users shortly after test starts. It then walks each user through the script assigned to it. The GET requests will be made in asynchronous bursts of up to 6 at a time such as HTML, images, JS, CSS, and XML. Each POST request will be made on its own to ensure that any dynamic response text from earlier requests was available for to be passed into the POST data. This also ensures that any response data from the POST is available for subsequent requests. When those responses are all returned, then the system pauses a random time between the Min & Max “Think Time”, then it requests the next page. This process continues until the end of the script. So, having more users going at the same time, without waiting more than a few hundred ms to get a response, will increase RPS.

     

  • If I stop a test before reaching the peak users, are the rest of my users wasted?

    You can terminate a test early because the target server fails. When this happens in a test using virtual users, those users may seem “wasted”.

    Customers have asked us why the system deducted 5,000 VUsers from their account when the test only lasted 10 minutes and stopped at 500 users. The reason LoadStorm does not prorate the VUsers to the minute or to the actual peak users is because we incur the full costs of the 5,000 VUser test.

    Amazon’s cloud is elastic, and we “rent” the servers needed for your tests. However, there is a minimum of 1 hour increments that Amazon will bill us. For example, if your test needs 20 servers to hit your defined peak users, we buy those 20 servers upfront before your test actually starts. Then if your load test dies 5 minutes into it, we are still charged as if the test used all 20 servers for 1 hour.

    Thus, we recommend that you start with smaller tests to verify that you have all of the environmental factors properly configured. Growing the volume in several tests that increase in volume will potentially eliminate the “wasted” feeling of throwing 5,000 users at a server that fails at 500 users.

     

  • How do I stop a test that has killed my server?

    You can press the “Stop Execution” button during a load test. To do this:

    1. click ANALYZE in the left navigation
    2. double-click the test run that is currently in progress
    3. near the top-right in the filters area is the “Stop Execution” button

    How to stop a load test in progress
     

  • Why aren’t the 500 users evenly distributed across all 5 of my servers behind a round robin load balancer?

    A round robin load balancer will not work very well for testing your configuration unless you are using a much higher number of virtual users (at least 1000 or more). Each of our simulation servers will resolve the DNS name independently of the others, but each of our servers will simulate hundreds of users. I would expect a peak load of 500 users to get routed to more than one of your servers, but probably not all 5 of them. A peak load of 200 users or less is likely to hit only one of your servers.

     

  • Will the load test show up in the Google Analytics reports? How is testing typically tracked by Google Analytics?

    We would not expect LoadStorm test volume to show up in your Google Analytics. LoadStorm does not execute the javascript, and requests outside of your domain are blocked unless those other servers have also been verified. Servers like Google Analytics should be set to “Ignore”. So, the tracking by Analytics, ad servers, or other systems outside of your domain would not work. Some of these kinds of services have restrictions against load testing.

    Almost all of our customers don’t want our load testing tool traffic to show up in their marketing statistics (Google Analytics) because it essentially renders their analytics data worthless. Additionally, Google has limits on the amount of hits per period in their free GA account; otherwise you must upgrade to the paying account. There are also legal issues involved whereby Google doesn’t want automated tools like LoadStorm hammering their applications (even GA) – it is a violation of their standard terms and conditions. Thus, when a GA tracking javascript is found in a page that is involved in a LoadStorm test, that server is ignored automatically by our tool.

    Most of our clients use a different mechanism for watching the user activity of a load test. There are many tools for monitoring the website, including several excellent options for open source and commercial products such as New Relic, Loggly, and Hyperic. Most of these are very inexpensive or free, up to a certain level. These types of tools provide significant information regarding the user activity as well as important data on your architecture. For example, is your web server memory bound when response times start to increase drastically? Or did your database server hit 99% utilization when throughput drops?

    New Relic and Hyperic can monitor server activity from the local server using an agent. Loggly is a tool that has teamed up with New Relic for analyzing server logs which can make your life easier when trying to extrapolate the user activity from your server log.

    So to sum it up any javascript that reports traffic to third-party analytic tools will not work with LoadStorm. Instead useful tools for monitoring server usage and analyzing log files of user activity on the server under test (SUT) our encouraged and will work with LoadStorm.

  • Can your system run load tests behind my firewall for non-Internet applications?

    Unfortunately not. LoadStorm is only for web applications that can be reached from the Amazon cloud. If you have an intranet application, then opening a hole in your firewall and providing a URL or public IP address that can reach the intranet application will allow LoadStorm to run load tests against it.

     

  • How much does LoadStorm emulate the caching abilities of a real browser client? Will cache be reset across various sessions?

    The short answer is: If your browser caches it, LoadStorm will cache it for a virtual user. When a virtual user completes a script it closes the session. Then it clears all cookies and cache before repeating the script.

    Our load testing tool is designed to closely mimic the browser, but it offers three settings for caching behavior. There are a few exceptions. We do not cache HTML resources since these are often dynamically generated.

    Three settings for caching behavior
     


Troubleshooting

  • I can’t run a load test because a server isn’t verified or ignored. Can you help me?

    Manage Servers

    Reach out to support@loadstorm.com for help with the verification process if you are stuck. For instructions on how to verify a server, please see our learning center‎ or watch this video tutorial.

    How does verification work?

    Our verification process works like Google Analytics. The idea is to prove it is your server so LoadStorm isn’t used for a denial-of-service attack. You can put an empty file that is named with a unique string generated by LoadStorm (i.e. loadstorm-<unique string>) into your root directory (doesn’t matter what is in it). So let’s say your target server is http://www.xyzcorp.com, then there should be a valid response if a request comes to your server for http://www.xyzcorp.com/loadstorm-90630fc588.html.

    Alternatively, if you don’t want to use the file you can embed the code in your default page using HTML comment tags. To continue the example, the HTML page received from requesting http://www.xyzcorp.com should contain a string within comment tags somewhere inside the document that matches “loadstorm-90630fc588″. LoadStorm will parse the page returned by your server exactly like a browser will. In that parsing, the verification string must appear somewhere for us to consider it verified.

    Example Code with a string in a comment tag:
    <html>
    <!– loadstorm-90630fc588 –>
    </html>

  • Why do I need to verify my server? How do I do that?

    This is a very common question. A example email may look something like this:

    I am unable to schedule a test run because LoadStorm shows that I need to verify a bunch of servers:

    which-servers-need-to-be-verified

    Click to zoom.

    I am not sure what to do now…Why do I have to verify these servers?

    Sincerely, Mr. WPL

    Are You Load Testing Facebook?

    From the selected script LoadStorm found that this test plan will be hitting servers that aren’t verified (authorized for testing). Put another way, these scripts contain requests that go to servers that do not belong to Mr. WPL in the example email, and LoadStorm gets in trouble when our customers load test other people’s servers without their permission.

    Does Mr. WPL really want to hit Facebook and Pinterest as part of his load test? If so, do those 3rd party servers contribute a crucial part of his server’s ability to handle load? If so, does he have their permission to hammer on their servers? Probably not. They tend to get upset when someone utilizes a cloud load testing tool to simulate thousands of VUsers against their social media platform. Please read the terms and conditions of use for those services.

    And where do those VUsers originate? LoadStorm. So who gets the phone call from their lawyers? We do.

    Prevent Denial of Service Attacks

    Our verification process is designed to prevent LoadStorm from being used as a Denial of Service (DOS) weapon. By placing a code on your home page that LoadStorm can request and confirm, our system confirms that you have the control over that server; thus, it is safe for our tool to hammer it with VUsers.

    Recommendation

    To verify a server follow the instructions in our learning center or watch this video tutorial.

    Ignore all third-party servers that are not crucial to your web application. LoadStorm will not make any requests to ignored servers. Remember the goal is to test the scalability of your web servers, and not those of third-parties. If you feel there is a third-party server that is crucial to your testing needs please email them and copy support@loadstorm.com requesting permission from their company to let you use our software to load test their web servers and yours as part of your testing needs.

  • Does LoadStorm use a full browser for accessing the websites to test or does it emulate a browser?

    load testing ramp time
    LoadStorm PRO does not use a browser emulator, nor is it a full browser. Instead, it takes a HAR recording and builds a script from the recorded network requests to mimic a real browser while still allowing for a competitively costed solution. LoadStorm PRO also supports AJAX requests as well as REST and SOAP API calls. If we used an actual browser, we would trade-off the cost effectiveness of our solution for the ability to actually process javascript and other interactions exactly as a browser. Some solutions use actual browsers and cost you a great deal more money.

    We understand some web developers need real browsers and are willing to pay for it. Our load testing tool is designed to help those developers that are willing to achieve their load testing goals without a full browser and at a much lower cost.

     

  • What are the IP address ranges used by LoadStorm? We need to white list them.

    Because we use the Amazon EC2 cloud to dynamically instantiate load generation servers we don’t have a specific static IP address. However, we can provide the possible ranges for each EC2 data center or geographic location that we use. The only groups of ranges that we do not use are GovCloud, and Beijing. The list of ranges are no longer updated from Amazon’s blog post about EC2 public IP ranges.

    Amazon now maintains the list of their EC2 public IP ranges in this JSON file:
    https://ip-ranges.amazonaws.com/ip-ranges.json

  • Other than IP address, is there any way to identify LoadStorm requests from normal traffic?

    Yes, there are two additional ways for you to white list our traffic:

    1. You can customize the user agent for each script which your firewall can see in the request headers of inbound requests. However, if your test scripts contains HTTPS requests, then those headers will be encrypted. Visit our learning center for more information on modifying a script’s default settings such as the User Agent.
    2. If you are not testing against your production server, we recommend trying a non-standard port number to identify the traffic as load testing. LoadStorm allows you to use any port number and can be designated for each target server.

    Thumbs Up Button

  • Can I create a test script where virtual users login to my web application? If so, how do I get my test user file into LoadStorm?

    Yes. Once you’ve created a script, you may modify form POSTs in order to log into your application with test user data.

    We recommend you visit our learning center, where we go over all of the options for parameterization in a script. A brief explanation is given below, but we also offer video tutorials on common examples such as parameterizing a login form.

    While creating a recording using developer tools in your browser, log into your application, and save the recording as a HAR file. Then upload the recording into your LoadStorm account. Open your script and select the parameterization tab.

    After that, click BUILD in the left navigation and switch to the User Data tab at the top. Here you can upload your CSV which contains necessary login info (username/email, passwords). Note that the first row’s data will be read as column headers and not row data. You can also choose to “generate data” on-the-fly in LoadStorm. If you need to embed a comma inside a particular data field, simply place quotation marks around all field data in the file.

    Finally, edit the necessary script and find the POST request that submits the form. Modify the form to include Custom Data with CSV user information. To add the test user data, you must select “custom” under the form modifications options, and select the data from a CSV file.

    This simulates traffic more realistically. Instead of simulating one user logging in over and over, many different users log in instead.

  • Can your system run load tests behind my firewall for non-Internet applications?

    Unfortunately not. LoadStorm is only for web applications that can be reached from the Amazon cloud. If you have an intranet application, then opening a hole in your firewall and providing a URL or public IP address that can reach the intranet application will allow LoadStorm to run load tests against it.

     

live chat service provider