Analyzing Test Results Archives – LoadStorm

Many of the errors will be HTTP status codes which can easily be looked up on wikipedia or RFC 7231. Some of the errors you encounter will not have an HTTP status code. These errors are defined by LoadStorm.

Common HTTP Status Codes

• 400: Bad Request – A VUser sent a request that is not properly formatted.
• 401: Unauthorized – A VUser failed authentication to access the target server.
• 403: Forbidden – A VUser lacked the permission to get a resource.
• 404: Not Found – Usually the link is invalid due to a typo, stale token, or the file does not exist.
• 500: Internal Server Error – Usually indicates the target server is having trouble with the load or something is wrong with the request.
• 502: Bad Gateway – Could be a configuration issue with target server.
• 503: Service Unavailable – Your server is barely functioning, and can only respond with this status code.

Common LoadStorm Error Codes

• Request Read Timeout – The request may have begun to respond but was cancelled when it reached the customizable timeout limit.
• Request Connection Timed Out – A VUser couldn’t establish a TCP connection to the target server after 10 seconds, and this limit is not adjustable.
• Socket Connection Reset – The connection was reset by the target server via a TCP reset packet which breaks the connection immediately.
• SSL Handshake Failure – The request was cancelled because it was unable to establish a secure connection possibly due to an obsolete cipher suite or mismatched SNI.

LoadStorm calculates the response time in the following manner:

The total time taken between the first byte of the request sent and the last byte of the response received for all requests made. This includes the network latency and the time for the server to respond.

The primary page normally includes the HTML, CSS, and Javascript files. However, the primary page could be another resource such as a PDF, video, or Flash file. The response time may appear slightly faster than observed by you in a browser because it does not include time for a browser to render the page. Of course render time will be affected by browser type, operating system, and horsepower of the client computer.

It is also important to note that LoadStorm’s servers probably have faster internet connections than many homes and offices.

LoadStorm is a truly distributed application that leverages the power of Amazon Web Services to scale on demand with processing power and bandwidth as needed to test the largest web projects. As you crank up the testing load from 200 to 1,000,000 virtual users, LoadStorm is automatically adding machines for you (as many as necessary) from Amazon’s data centers to handle the processing. When your tests are done and the extra machines are not needed, they are turned off to wait for another test.

A script is processed as a sequential series of pages that you have defined in the recording. Once a page finishes, the VUser pauses for the specified time (between Min & Max “Think Time” in the script), then the next page proceeds. After the last page completes, the VUser context (session, cache, cookies) is cleared and the script starts from the beginning again with the same VUser. This same VUser will use different data upon script repeat if a CSV of User Data or dynamic response text has been parameterized within the script. As the concurrent users scale up they each begin the script for the first time, and also repeat the script after each completion.

You define the number of concurrent users you want at the start and peak of the test. You can also define a ramp down period to see how well your system recovers as the traffic subsides.

Scripts are repeated until the test is done as defined by the test duration field (in minutes) from when you began the load test.

You can press the “Stop Execution” button during a load test. To do this:

1. click ANALYZE in the left navigation
2. double-click the test run that is currently in progress
3. near the top-right in the filters area is the “Stop Execution” button

If one of your web pages contains a reference to an image that is not available on your system, then a response header will be sent to LoadStorm with Status Code = 404. Additionally, your web server will send an HTML page indicating the 404 “Not Found” page to alert the user. That’s standard practice in web applications.

The reason LoadStorm reports the 404 errors as HTML is because your web server is sending our system the Content Type = “HTML” attribute. LoadStorm separates the data by the Content Type.

A round robin load balancer will not work very well for testing your configuration unless you are using a much higher number of virtual users (at least 1000 or more). Each of our simulation servers will resolve the DNS name independently of the others, but each of our servers will simulate hundreds of users. I would expect a peak load of 500 users to get routed to more than one of your servers, but probably not all 5 of them. A peak load of 200 users or less is likely to hit only one of your servers.

We would not expect LoadStorm test volume to show up in your Google Analytics. LoadStorm does not execute the javascript, and requests outside of your domain are blocked unless those other servers have also been verified. Servers like Google Analytics should be set to “Ignore”. So, the tracking by Analytics, ad servers, or other systems outside of your domain would not work. Some of these kinds of services have restrictions against load testing.

Almost all of our customers don’t want our load testing tool traffic to show up in their marketing statistics (Google Analytics) because it essentially renders their analytics data worthless. Additionally, Google has limits on the amount of hits per period in their free GA account; otherwise you must upgrade to the paying account. There are also legal issues involved whereby Google doesn’t want automated tools like LoadStorm hammering their applications (even GA) – it is a violation of their standard terms and conditions. Thus, when a GA tracking javascript is found in a page that is involved in a LoadStorm test, that server is ignored automatically by our tool.

Most of our clients use a different mechanism for watching the user activity of a load test. There are many tools for monitoring the website, including several excellent options for open source and commercial products such as New Relic, Loggly, and Hyperic. Most of these are very inexpensive or free, up to a certain level. These types of tools provide significant information regarding the user activity as well as important data on your architecture. For example, is your web server memory bound when response times start to increase drastically? Or did your database server hit 99% utilization when throughput drops?

New Relic and Hyperic can monitor server activity from the local server using an agent. Loggly is a tool that has teamed up with New Relic for analyzing server logs which can make your life easier when trying to extrapolate the user activity from your server log.

So to sum it up any javascript that reports traffic to third-party analytic tools will not work with LoadStorm. Instead useful tools for monitoring server usage and analyzing log files of user activity on the server under test (SUT) our encouraged and will work with LoadStorm.

The short answer is: If your browser caches it, LoadStorm will cache it for a virtual user. When a virtual user completes a script it closes the session. Then it clears all cookies and cache before repeating the script.

Our load testing tool is designed to closely mimic the browser, but it offers three settings for caching behavior. There are a few exceptions. We do not cache HTML resources since these are often dynamically generated.

The starting and peak number of virtual users in LoadStorm represent concurrent users or the number of simulated users at a particular point in time. Each concurrent user will last for the duration of the script. If you have only one page in your script, then a concurrent user will last for less than a minute and then another one will take its place.

For example, if you have a test with one script having one page, starting at 10 concurrent users and peak of 10 concurrent users for 20 minutes, there will be 200-300 total users simulated (and same number of sessions). LoadStorm will create a new session for each virtual user, but it will not maintain the session after the script has ended, so your session should time out as you have configured it.

The LoadStorm test parameters simulate “concurrent” users not total users. The total simulated users will generally be much higher than concurrent users, but the actual count depends on lots of other factors such as the number of pages in the script, think times, etc.

The short answer is no. LoadStorm reporting graphs include a metric called Requests Per Second, and that is what you are describing.

A virtual or simulated user does what a “real” user does as specified by the scripts and pages that you have created in LoadStorm. If there are X number of users, then there are X script instances running at that particular time.

When creating a script, you can adjust the “Think Times” which is how much time there is after each page. The actual pause is randomly generated to give a more realistic load against the server. As an example, a script which makes a single request with an average think time of 30 seconds will have an average of 2 requests per minute per user. It gets more complicated when there are many GET requests like Javascript, CSS, and images. LoadStorm makes up to six GET requests at a time asynchronously so as soon as one of the first six completes another starts. This means that each response time has an impact on the number of requests per second.

For a nice visual of how this works see the Waterfall tab within your script: