All FAQ Archives – LoadStorm

• What do these errors mean?

  Many of the errors will be HTTP status codes which can easily be looked up on wikipedia or RFC 7231. Some of the errors you encounter will not have an HTTP status code. These errors are defined by LoadStorm.

  Common HTTP Status Codes

  • 400: Bad Request – A VUser sent a request that is not properly formatted.
  • 401: Unauthorized – A VUser failed authentication to access the target server.
  • 403: Forbidden – A VUser lacked the permission to get a resource.
  • 404: Not Found – Usually the link is invalid due to a typo, stale token, or the file does not exist.
  • 500: Internal Server Error – Usually indicates the target server is having trouble with the load or something is wrong with the request.
  • 502: Bad Gateway – Could be a configuration issue with target server.
  • 503: Service Unavailable – Your server is barely functioning, and can only respond with this status code.

  Common LoadStorm Error Codes

  • Request Read Timeout – The request may have begun to respond but was cancelled when it reached the customizable timeout limit.
  • Request Connection Timed Out – A VUser couldn’t establish a TCP connection to the target server after 10 seconds, and this limit is not adjustable.
  • Socket Connection Reset – The connection was reset by the target server via a TCP reset packet which breaks the connection immediately.
  • SSL Handshake Failure – The request was cancelled because it was unable to establish a secure connection possibly due to an obsolete cipher suite or mismatched SNI.
• How can I use analytics to estimate the number of virtual users to ramp up to during my load test?

  That’s a good question. Nowadays tools like Google Analytics can offer real time monitoring which will give you the best insight into your peak levels of traffic and how that traffic may spike for a few minutes. If you have this information it already provides a good estimate for many peak users you should test for in LoadStorm.

  Most commonly though Google Analytics will have data on the number of visits per hour to your site as well as the average duration of those visits. This makes it a bit harder to estimate what volume of users you should test for. With this information we can estimate the average concurrent visitors that were on your site using the following equation:

  U = V / (60/D)

  Where: U is the number of load test virtual users (that’s what we are trying to figure out) V is the average number of visitors per hour D is the average duration of a visitor

  60 is the number of minutes in an hour

Let’s state this formula again in English like a math word problem:

Load Test Virtual Users is equal to the Average Visitors per Hour divided by the User Turnover Rate per Hour.

Ideally you should test for an amount that is a bit greater than this estimate just to make sure you have a cushion for unusual spikes in traffic. For a more lengthy explanation please visit our blog post about virtual user calculations.

• Can LoadStorm handle WebSockets?

  Unfortunately we do not have a method yet for handling the WebSocket protocol. LoadStorm can still make the HTTP request that would initialize the handshake for the WebSocket connection, but no connection will be made. There are two issues that arise with WebSockets. Handling the non-HTTP protocol, and deciding when to terminate the connection. The connection usually remains open until the visitor has completed some action or prematurely departs the site.

• Why did my QuickStorm fail?

  Our QuickStorm feature is meant to help give a quick demonstration of what our reporting will look like during a test run. To make this process as smooth as possible we remove the task of having you make a browser recording, saving it as a HTTP Archive (HAR) file, uploading it to LoadStorm to be converted into a script and scheduling a test run. Instead we take a URL you provide and feed it into an automated process that handles these steps for you. Anytime you try to feed in complex elements into an automated process there are bound to be some issues that may prevent the process from completing.

  Sometimes these are simple mistakes:

  • typo mistakes in the URL
  • requesting URLs using HTTP when they need HTTPS
  • private URLs
  • local IP addresses
  • URLs that need basic authentication credentials

  Other times there are more complex issues with the automated tool handling the page given to it. Usually this is due to how it fails to parse something in the page:

  • handling WebSockets
  • malformed html
  • javascript errors
  • streaming audio/video
  • failing to make a connection for a requested resource

  Many of these are not an issue with real browser recordings. So if you encounter a problem please use the LiveChat button at the top of the QuickStorm test and ask for assistance, or email [email protected] to request a demo. Otherwise we encourage you to check out this guide for getting started with LoadStorm to create a real browser recording and run your first load test instead of a QuickStorm.

• What does LoadStorm do?

  
  LoadStorm™ is a web-based load testing tool for simulating what users do with a web site or web application. You use it to build tests that send requests to your server in the same way that a user’s browser sends requests to your server. But these tests are executed by our automated systems rather than by a user, so they can be done repeatedly and in large numbers simultaneously. They can also be built using our tool in such a way as to simulate a large number of different users with different tasks to perform.

• How much does it cost?

  
  It depends on how many virtual users you need for testing, and it depends on if you would like to use our consulting services.

  Creating an account is free, but a free account is limited to 50 virtual users to try out our product. By default, we offer four pricing plans. Each plan scales up in cost, but offers more benefits such as bundled consulting hours.

  For more details, please see Load Testing Cost.

• Do I need to manage my servers? How do I do that?

  Depending on the test requirements, third-party servers should be set to “Ignored” or “Archived”. Requests that would go to ignored or archived servers will be skipped instead. Skipped requests prevent unnecessary traffic to advertisements, analytics, or other third-party services. You may encounter a server already set to ignored as a part of our blacklist. Please contact [email protected] about enabling a blacklisted server for your testing needs.

  Ignore a Server

  Please note that by ignoring a server, any requests normally sent to that server will not be sent.

  1. Click on the Build menu, then on the Server tab on the top.
  2. Double-click the server you want to ignore.
  3. Click the Ignore button. No requests will be sent to this server.

  Archive a Server

  Archiving a server works the same as ignoring a server, but it’s a separate category. This way you can separate the servers you may never need to test from servers that you may not want to test at the moment. To reiterate, archiving a server means it is ignored. Any requests normally sent to that server will not be sent.

  1. Click on the Build menu, then on the Server tab on the top.
  2. Click the server you want to archive to select it.
  3. Click the Archive button. No requests will be sent to this server.

  

• How should I set up scripts to simulate real traffic?

  Most people create several scripts to accurately reflect different types of users. Scripts execute simultaneously in order to hit your site with the correct mix of traffic. You control the weighting of each script by giving it a number relative to the others. For example, if you have 2 scripts and you give one a weighting of 4 and the other a weighting of 1, then the first will generate 80% of the traffic and the second 20%. This 80/20 could be realistic for a retail e-commerce site where most of the people (80%) are browsing the product catalog, while a minority of people (20%) are going through a buying experience. You can of course throw in a low-weight scenario to represent some internal employees adding products or issuing refunds.

  Another way of entering the weight of the traffic is to just enter the exact percentage you would like each script to carry. For the example above you would simply enter 80 and 20. The only thing you have to be very careful about is that the percentages add up to 100. Otherwise, it will work like the above description, which is different.

  There is no limit to the number of scripts you can create, and you can use many scripts in a test. We encourage you to build as many as necessary to accurately reflect the real traffic patterns on your site. That will give you the most meaningful metrics for tuning your applications.

  We also simulate the user “think time” with random pauses after each page. Real users click on something, then read the page or fill in a form, and then click on something else. Thus, you don’t want to have a simulated virtual user clicking on something every .5 seconds – it would not give you meaningful results if you want to see how your site holds up in the real world.

  Also see: Virtual User Calculations about calculating the amount of virtual users needed for a realistic test.

• Why should I ramp up my test load volume over time?

  It helps with performance engineering. We recommend to not begin a large test at the maximum number of virtual users. Rather than run a test for 10 minutes beginning with 5,000 and ending with 5,000, we suggest to start smaller, for example, with 500 users and increase to 5,000 over an hour. This approach has the advantage of allowing your system to get all the parts working properly at smaller load (e.g. caches, threads, database connections) before the heavier volume starts exposing potential bottlenecks in your application. Also if the target server begins to fail during the test this provides more granularity into the number of concurrent users where failure begins. If the test starts at 5,000 and fails all that can be learned from the results is that it fails somewhere between 1 and 5,000 which isn’t very useful.

  From what we’ve seen in thousands of load tests with LoadStorm, it is common to see useful patterns in the metrics before the peak traffic is reached, and these metrics point to areas of performance limitations that can be tuned.

  Load testing is invariably coupled with performance tuning – an iterative process of test/tune/test/tune. Therefore, we recommend you ramp the volume in order to get more value from the test results.

• Why doesn’t my test start immediately?

  LoadStorm needs a little time to prepare the test resources such as the load generation servers. Our system is using each selected script(s) to pre-calculate the virtual users’ actions, how many servers will need to be instantiated from the cloud, random think times, etc. This produces essentially a test roadmap that our Scheduler and Summarizer modules use to coordinate all the HTTP traffic and correctly capture the metrics. Depending on the number of concurrent users and the duration of your test, the preparation of a load test can take 2 to 15 minutes.

• Does LoadStorm use a full browser for accessing the websites to test or does it emulate a browser?

  
  LoadStorm PRO does not use a browser emulator, nor is it a full browser. Instead, it takes a HAR recording and builds a script from the recorded network requests to mimic a real browser while still allowing for a competitively costed solution. LoadStorm PRO also supports AJAX requests as well as REST and SOAP API calls. If we used an actual browser, we would trade-off the cost effectiveness of our solution for the ability to actually process javascript and other interactions exactly as a browser. Some solutions use actual browsers and cost you a great deal more money.

  We understand some web developers need real browsers and are willing to pay for it. Our load testing tool is designed to help those developers that are willing to achieve their load testing goals without a full browser and at a much lower cost.

• What are the IP address ranges used by LoadStorm? We need to white list them.
• Other than IP address, is there any way to identify LoadStorm requests from normal traffic?

  Yes, there are two additional ways for you to white list our traffic:

  1. You can customize the user agent for each script which your firewall can see in the request headers of inbound requests. However, if your test scripts contains HTTPS requests, then those headers will be encrypted. Visit our learning center for more information on modifying a script’s default settings such as the User Agent.
  2. If you are not testing against your production server, we recommend trying a non-standard port number to identify the traffic as load testing. LoadStorm allows you to use any port number and can be designated for each target server.

  

• What does “response time” mean?

  
  LoadStorm calculates the response time in the following manner:

  The total time taken between the first byte of the request sent and the last byte of the response received for all requests made. This includes the network latency and the time for the server to respond.

  The primary page normally includes the HTML, CSS, and Javascript files. However, the primary page could be another resource such as a PDF, video, or Flash file. The response time may appear slightly faster than observed by you in a browser because it does not include time for a browser to render the page. Of course render time will be affected by browser type, operating system, and horsepower of the client computer.

  It is also important to note that LoadStorm’s servers probably have faster internet connections than many homes and offices.

• Can I create a test script where virtual users login to my web application? If so, how do I get my test user file into LoadStorm?

  Yes. Once you’ve created a script, you may modify form POSTs in order to log into your application with test user data.

  We recommend you visit our learning center, where we go over all of the options for parameterization in a script. A brief explanation is given below, but we also offer video tutorials on common examples such as parameterizing a login form.

  While creating a recording using developer tools in your browser, log into your application, and save the recording as a HAR file. Then upload the recording into your LoadStorm account. Open your script and select the parameterization tab.

  After that, click BUILD in the left navigation and switch to the User Data tab at the top. Here you can upload your CSV which contains necessary login info (username/email, passwords). Note that the first row’s data will be read as column headers and not row data. You can also choose to “generate data” on-the-fly in LoadStorm. If you need to embed a comma inside a particular data field, simply place quotation marks around all field data in the file.

  Finally, edit the necessary script and find the POST request that submits the form. Modify the form to include Custom Data with CSV user information. To add the test user data, you must select “custom” under the form modifications options, and select the data from a CSV file.

  This simulates traffic more realistically. Instead of simulating one user logging in over and over, many different users log in instead.

• How does it generate HTTP traffic against my web application?

  LoadStorm is a truly distributed application that leverages the power of Amazon Web Services to scale on demand with processing power and bandwidth as needed to test the largest web projects. As you crank up the testing load from 200 to 1,000,000 virtual users, LoadStorm is automatically adding machines for you (as many as necessary) from Amazon’s data centers to handle the processing. When your tests are done and the extra machines are not needed, they are turned off to wait for another test.

  A script is processed as a sequential series of pages that you have defined in the recording. Once a page finishes, the VUser pauses for the specified time (between Min & Max “Think Time” in the script), then the next page proceeds. After the last page completes, the VUser context (session, cache, cookies) is cleared and the script starts from the beginning again with the same VUser. This same VUser will use different data upon script repeat if a CSV of User Data or dynamic response text has been parameterized within the script. As the concurrent users scale up they each begin the script for the first time, and also repeat the script after each completion.

  You define the number of concurrent users you want at the start and peak of the test. You can also define a ramp down period to see how well your system recovers as the traffic subsides.

  Scripts are repeated until the test is done as defined by the test duration field (in minutes) from when you began the load test.

• How Can I Increase the Requests per Second in My Test?

  Here are some thoughts on increasing RPS in LoadStorm tests:

  1. Reducing the think time added after each page in a script; the minimum pause is 5 seconds, so set both min & max to 5 seconds.
  2. Increasing the number of concurrent users.
  3. Reducing the response time from your server which decreases the time LoadStorm has to wait for a request to complete.
  4. Compressing the size of the response is one way to decrease the time it takes for the request to complete as well as save on throughput.

  RPS will always be impacted by the types of pages/resources your scripts are requesting. The way it works is that LoadStorm cranks up the starting concurrent users shortly after test starts. It then walks each user through the script assigned to it. The GET requests will be made in asynchronous bursts of up to 6 at a time such as HTML, images, JS, CSS, and XML. Each POST request will be made on its own to ensure that any dynamic response text from earlier requests was available for to be passed into the POST data. This also ensures that any response data from the POST is available for subsequent requests. When those responses are all returned, then the system pauses a random time between the Min & Max “Think Time”, then it requests the next page. This process continues until the end of the script. So, having more users going at the same time, without waiting more than a few hundred ms to get a response, will increase RPS.

• Can I change the host file for my test? I need to hit a server that does NOT currently have a valid DNS name because we don’t want it found by customers until we launch the site.

  Unfortunately, our load testing tool doesn’t have a feature for what you need. Normally this is a simple process of putting a domain name to IP address mapping in a configuration file on a particular machine. The process is not as easy in our situation because we use dynamically instantiated servers from the Amazon cloud; thus, we do not know what IP addresses we will be using before you run your tests. We would need to create new images specifically for your tests, and we would need to make changes to our core launching software that instantiates the images so that new machines are launched with this particular host mapping. If you would like us to coordinate this custom process, please contact our support. Additional charges per test will apply.

• If I stop a test before reaching the peak users, are the rest of my users wasted?

  You can terminate a test early because the target server fails. When this happens in a test using virtual users, those users may seem “wasted”.

  Customers have asked us why the system deducted 5,000 VUsers from their account when the test only lasted 10 minutes and stopped at 500 users. The reason LoadStorm does not prorate the VUsers to the minute or to the actual peak users is because we incur the full costs of the 5,000 VUser test.

  Amazon’s cloud is elastic, and we “rent” the servers needed for your tests. However, there is a minimum of 1 hour increments that Amazon will bill us. For example, if your test needs 20 servers to hit your defined peak users, we buy those 20 servers upfront before your test actually starts. Then if your load test dies 5 minutes into it, we are still charged as if the test used all 20 servers for 1 hour.

  Thus, we recommend that you start with smaller tests to verify that you have all of the environmental factors properly configured. Growing the volume in several tests that increase in volume will potentially eliminate the “wasted” feeling of throwing 5,000 users at a server that fails at 500 users.

• Why not let our users test the system’s capacity?

  For some projects, this is an acceptable approach. For some internal applications, apps that are very simple will realistically expect light traffic. The best plan may be to get it done as quickly as possible. You have to ask yourself what is the risk of the application not working at some point. For many businesses, getting a large influx of new users is the peak of success, and that success would be shattered by having the new users unable to use the system, unable to buy your product and unhappy with your company.

• How do I stop a test that has killed my server?

  You can press the “Stop Execution” button during a load test. To do this:

  1. click ANALYZE in the left navigation
  2. double-click the test run that is currently in progress
  3. near the top-right in the filters area is the “Stop Execution” button

  
   

• Why does the Summary Table of test results show my missing JPGs in the HTML row?

  If one of your web pages contains a reference to an image that is not available on your system, then a response header will be sent to LoadStorm with Status Code = 404. Additionally, your web server will send an HTML page indicating the 404 “Not Found” page to alert the user. That’s standard practice in web applications.

  The reason LoadStorm reports the 404 errors as HTML is because your web server is sending our system the Content Type = “HTML” attribute. LoadStorm separates the data by the Content Type.

• Why aren’t the 500 users evenly distributed across all 5 of my servers behind a round robin load balancer?

  A round robin load balancer will not work very well for testing your configuration unless you are using a much higher number of virtual users (at least 1000 or more). Each of our simulation servers will resolve the DNS name independently of the others, but each of our servers will simulate hundreds of users. I would expect a peak load of 500 users to get routed to more than one of your servers, but probably not all 5 of them. A peak load of 200 users or less is likely to hit only one of your servers.

• Will the load test show up in the Google Analytics reports?

  We would not expect LoadStorm test volume to show up in your Google Analytics. LoadStorm does not execute the javascript, and requests outside of your domain are blocked unless those other servers have also been verified. Servers like Google Analytics should be set to “Ignore”. So, the tracking by Analytics, ad servers, or other systems outside of your domain would not work. Some of these kinds of services have restrictions against load testing.

  Almost all of our customers don’t want our load testing tool traffic to show up in their marketing statistics (Google Analytics) because it essentially renders their analytics data worthless. Additionally, Google has limits on the amount of hits per period in their free GA account; otherwise you must upgrade to the paying account. There are also legal issues involved whereby Google doesn’t want automated tools like LoadStorm hammering their applications (even GA) – it is a violation of their standard terms and conditions. Thus, when a GA tracking javascript is found in a page that is involved in a LoadStorm test, that server is ignored automatically by our tool.

  Most of our clients use a different mechanism for watching the user activity of a load test. There are many tools for monitoring the website, including several excellent options for open source and commercial products such as New Relic, Loggly, and Hyperic. Most of these are very inexpensive or free, up to a certain level. These types of tools provide significant information regarding the user activity as well as important data on your architecture. For example, is your web server memory bound when response times start to increase drastically? Or did your database server hit 99% utilization when throughput drops?

  New Relic and Hyperic can monitor server activity from the local server using an agent. Loggly is a tool that has teamed up with New Relic for analyzing server logs which can make your life easier when trying to extrapolate the user activity from your server log.

  So to sum it up any javascript that reports traffic to third-party analytic tools will not work with LoadStorm. Instead useful tools for monitoring server usage and analyzing log files of user activity on the server under test (SUT) our encouraged and will work with LoadStorm.

• Can your system run load tests behind my firewall for non-Internet applications?

  Unfortunately not. LoadStorm is only for web applications that can be reached from the Amazon cloud. If you have an intranet application, then opening a hole in your firewall and providing a URL or public IP address that can reach the intranet application will allow LoadStorm to run load tests against it.

• How much does LoadStorm emulate the caching abilities of a real browser client? Will cache be reset across various sessions?

  The short answer is: If your browser caches it, LoadStorm will cache it for a virtual user. When a virtual user completes a script it closes the session. Then it clears all cookies and cache before repeating the script.

  Our load testing tool is designed to closely mimic the browser, but it offers three settings for caching behavior. There are a few exceptions. We do not cache HTML resources since these are often dynamically generated.

  
   

• What’s the difference between “Concurrent Users”, “Sessions”, and “Total Users”?

  The starting and peak number of virtual users in LoadStorm represent concurrent users or the number of simulated users at a particular point in time. Each concurrent user will last for the duration of the script. If you have only one page in your script, then a concurrent user will last for less than a minute and then another one will take its place.

  For example, if you have a test with one script having one page, starting at 10 concurrent users and peak of 10 concurrent users for 20 minutes, there will be 200-300 total users simulated (and same number of sessions). LoadStorm will create a new session for each virtual user, but it will not maintain the session after the script has ended, so your session should time out as you have configured it.

  The LoadStorm test parameters simulate “concurrent” users not total users. The total simulated users will generally be much higher than concurrent users, but the actual count depends on lots of other factors such as the number of pages in the script, think times, etc.

• How many users and how many requests are hitting my web app?

  The short answer is no. LoadStorm reporting graphs include a metric called Requests Per Second, and that is what you are describing.

  A virtual or simulated user does what a “real” user does as specified by the scripts and pages that you have created in LoadStorm. If there are X number of users, then there are X script instances running at that particular time.

  When creating a script, you can adjust the “Think Times” which is how much time there is after each page. The actual pause is randomly generated to give a more realistic load against the server. As an example, a script which makes a single request with an average think time of 30 seconds will have an average of 2 requests per minute per user. It gets more complicated when there are many GET requests like Javascript, CSS, and images. LoadStorm makes up to six GET requests at a time asynchronously so as soon as one of the first six completes another starts. This means that each response time has an impact on the number of requests per second.

  For a nice visual of how this works see the Waterfall tab within your script: