Analyzing Test Results Archives | Page 2 of 2 | LoadStorm

Average Duration is the time taken for ALL steps to complete (see step duration calculation), plus the average pause times between steps. Thus, it is the mean time for a specific vuser’s “life”. The actual duration fluctuates with the amount of pause assigned to that vuser. The Average Duration is shown above the steps on the Scenario page.

Duration is the time taken for a step to complete, which is the sum of time to last byte for each request in the step (including HTML, JS, CSS, images, etc.). The Average Duration shown above the steps on the Scenario page is the total of all step durations plus the average pause times between steps.

In LoadStorm the response time is calculated thus: The total time taken between the first byte of the request sent and the last byte of the response received for all requests made. This includes the network latency and the time for the server to respond.

The primary page normally includes the HTML, CSS, and Javascript files. However, the primary page could be another resource such as a PDF, video, or Flash file. The response time may appear slightly faster than observed by you in a browser because it does not include time for a browser to render the page. Of course render time will be affected by browser type, operating system, and horsepower of the client computer.

It is also important to note that LoadStorm’s servers probably have faster internet connections than many homes and offices.

LoadStorm is a truly distributed application that leverages the power of Amazon Web Services to scale on demand with processing power and bandwidth as needed to test the largest web projects. As you crank up the testing load to 200 hundred or 20,000 virtual users, LoadStorm is automatically adding machines for you (as many as necessary) from Amazon’s server farm to handle the processing. When your tests are done and the extra machines are not needed, they are turned off to wait for another test.

The normal processing is for a scenario to be processed as a sequential series of steps that you have defined in the test plan. Once a step finishes, the vuser pauses for the specified time (between min and max for scenario), then the next step proceeds. After the last step completes, the vuser context (session, cache, cookies) is cleared and the scenario starts from the beginning again with a new vuser.

You define the number of concurrent users you want, start and peak, during your test. As one vuser completes the scenario as described above, then another vuser is created to replace it – starting at step 1 again.

If you have started at a small number and set the peak number much higher, then LoadStorm will not only retire vusers that have completed the scenario, it will also add additional users throughout the test to ramp up the volume.
This is repeated until the test is done as defined by the test duration field (in minutes) when you began the load test.

If one of your web pages contains a reference to an image that is not available on your system, then a response header will be sent to LoadStorm have Status Code = 404. Additionally, your web server will send an HTML page indicating the 404 “Not Found” page to alert the user. That’s standard practice in web applications.

The reason LoadStorm reports the 404 errors as HTML is because your web server is sending our system the Content Type = “HTML” attribute. LoadStorm separates the data in the HTML and Other rows by the Content Type.

A round robin load balancer will not work very well for testing your configuration unless you are using a much higher number of virtual users (at least 2000 or so). Each of our simulation servers will resolve the DNS name independently of the others, but each of our servers will simulate hundreds of users. I would expect a peak load of 500 users to get routed to more than one of your servers, but probably not all 5 of them. A peak load of 100 users or less is likely to hit only one of your servers.

We would not expect LoadStorm test volume to show up in your Google Analytics. We do execute the javascript, but requests outside of your domain are blocked unless those other servers have also been verified. Servers like Google Analytics should be set to “Ignore”. So, the tracking by Analytics, ad servers or other systems outside of your domain would not work. Some of these kinds of services have restrictions against load testing.

Almost all of our customers don’t want our load testing tool traffic to show up in their marketing statistics (Google Analytics) because it essentially renders their analytics data worthless. Additionally, Google has limits on the amount of hits per period in their free GA account; otherwise you must upgrade to the paying account. There are also legal issues involved whereby Google doesn’t want automated tools like LoadStorm hammering their applications (even GA) – it is a violation of their standard terms and conditions. Thus, when a GA tracking Javascript is found in a page that is involved in a LoadStorm test, that server is ignored automatically by our tool.

Most of our clients use a different mechanism for watching the user activity of a load test. There are many tools for monitoring the website, including several excellent options for open source and commercial products such as New Relic and Hyperic. Most of these are very inexpensive or free up to a certain level. These types of tools provide significant information regarding the user activity as well as important data on your architecture. For example, is your web server memory bound when response times start to increase drastically? Or did your database server hit 99% utilization when throughput drops?

The short answer is: If your browser caches it, LoadStorm will cache it for a virtual user. When a virtual user completes a scenario, it is retired which causes the session to end.

Our load testing tool is designed to closely emulate the browser with a new, empty cache between sessions. So, generally your static assets will be cached for later steps in a scenario according to their header values. There are a few exceptions. We do not cache HTML resources since these are usually dynamically generated. Most LoadStorm users expect a subsequent HTML request to get this file again, thus that is how we coded it to work. If the headers are set properly, this will usually happen anyhow.

The starting and peak number of virtual users in LoadStorm represent concurrent users or the number of simulated users at a particular point in time. Each concurrent user will last for the duration of the scenario. If you have only one step in your scenario, then a concurrent user will last for less than a minute and then another one will take its place.

For example, if you have a test with one scenario having one step, starting at 10 concurrent users and peak of 10 concurrent users for 20 minutes, there will be 200-300 total users simulated (and same number of sessions). LoadStorm will create a new session for each virtual user, but it will not maintain the session after the scenario has ended, so your session should time out as you have configured it.

The LoadStorm test parameters simulate “concurrent” users not total users. The total simulated users will generally be much higher than concurrent users, but the actual count depends on lots of other factors such as the number of steps in the scenarios, pause times, etc.

Page views is a good term when reviewing Google Analytics data for traffic analysis. However, when reviewing load test data, page views can be a confusing metric. We do not track page views; rather we track requests and responses. Let’s look at the difference and how LoadStorm measures interactions with the server.

The page views per virtual user should stay consistent as long as the response time is consistent and no errors. In some tests, the response time and errors are significant and increase over the duration of the test. For example, we see some load tests that have a large number of timeouts (no response to the request in 35 seconds). That means that LoadStorm cannot move to the next request for a virtual user until the response is received or we timeout the request. This drastically changes the equation you may be using to calculate the number of pageviews. Specifically, in one test there were 143,553 errors out of 1,345,534 total requests during a 30 minute test. This failure rate greatly decreased the number of requests LoadStorm was sending during the test.

Also, if a request is sent for a step, LoadStorm is using the response header to capture the content type. A normal HTML page in a test returns, “text/html; charset=UTF-8″. When no response is received, LoadStorm assigns the response content type as, “NONE”. This affects the summary report table because the NONE responses are lumped into the OTHER category rather than the HTML category. Therefore, many of the 143k errors are shown in the OTHER category – not in the HTML category. That means that many of the HTML requests are not even shown in the table as HTML (or page views).

The bottom line is that it’s mathematically invalid to expect linear correlation between page views, requests/responses, and vusers if the target server is not responding well. Everyone has a different definition of test success and failure. Some believe that greater than 1% error rate is a reason to stop the test. Others believe that an average response time greater than 3 seconds is a failure. Regardless how you define failure, timeouts and other server errors degrade the value of your load test metrics.

The short answer is no. LoadStorm reporting graphs include a metric called Requests Per Second, and that is what you are describing.

A virtual or simulated user does what a “real” user does as specified by the scenarios and steps that you have created in LoadStorm. If there are X number of users, then there are X scenarios running at that particular time.

When creating a scenario, you can adjust the pause times which is how much time there is between steps. The actual pause is randomly generated to give a more realistic load against the server. As an example, a scenario which opens a single page for each step with an average pause of 30 seconds will have an average of 2 requests per minute per user. It gets a bit more complicated when there are external files like Javascript. These are added as well and counted in the total requests. When one scenario ends for a designated number of users, another is started within a minute to maintain the desired user load until the end of the test.

The vuser continues to move to the next step in its scenario. For instance, if a JavaScript file fails to download during step 3:

• That error will be logged in LoadStorm for reporting
• That request will be terminated
• All other requests for step 3 will be attempted as usual
• The vuser will begin step 4 as defined in the scenario

If any particular resource times out within a step, some of the step processing may not finish. This depends on what kind of resource fails and at what point in the step. If an image fails, this is relatively harmless. The images following will not be requested, but everything else will continue normally. If the html fails with a timeout, then the step will stop and no images or other resources will be requested. Images, javascript and css all depend on the html, so none will be requested for that step unless the html is received without error.

The step following a step with an error will work normally to the extent that it can. As with a browser, you cannot click on a link or submit a form if you do not currently have a valid page available with which to work. You can always type in a new url and go from there. Likewise, in LoadStorm if the next step is an open step, then previous steps will have no effect except for possibly login or cookie state. If the next step is a click or form, then the previous html must have been received for the step to function. If the step cannot work because the previous html timed out or otherwise failed, then that step will show no requests or errors and the next step will be attempted after the expected pause.

There is currently only one situation where a scenario will restart before all of its steps have run. This is a step timeout where a step starts but does not finish before 5 minutes have passed. A step includes getting all resources within the step such as html, javascript, images, css, etc. So, it is not inconceivable that this timeout occurs if there are 50, 100 or more resources in a step, but it is long enough that it is pretty rare.

LoadStorm creates the starting concurrent users within seconds of the test starting. For example, if you set the test to start at 1,000 users then you will have requests coming from those 1,000 users in the first 30 seconds. It then processes each virtual user through the scenario assigned to it. The first step is executed such as requesting your home page. The HTML is returned from your server and LoadStorm parses it into the DOM. All of the other resources are determined (images, Javascript, CSS, and XML) are requested separately and sequentially.

When those responses are all returned, then the system pauses a random time between min & max pause (you control this), then it requests the next step. That continues until the virtual user reaches the end of the scenario, whereupon that vuser is “retired”. After about 30 seconds, another vuser is created and the process starts again. The cache for the new virtual user starts empty and does not remember cookies or static resources for the previous vuser.