- I can’t run a load test because a server isn’t verified or ignored. Can you help me?
We can help with the verification process if you are stuck. Please see our learning center for how to Verify a Server.
Another option is to Ignore the server. That simply means that LoadStorm will not send requests to it. By ignoring a server, your test may continue and generate load against your servers without making unnecessary requests to external sites such as Facebook, Twitter, or even Google Analytics.
How does verification work?
Our verification process works exactly like Google Analytics. The idea is to prove it is your server so LoadStorm isn’t used for a denial-of-service attack. You can put an empty file that is named with a unique string generated by LoadStorm (i.e. loadstorm-) into your root directory (doesn’t matter what is in it). So let’s say your target server is http://www.xyzcorp.com, then there should be a valid response if a request comes to your server for http://www.xyzcorp.com/loadstorm-90630fc588.html.
Alternatively, if you don’t want to use the file you can embed the code in your default page using comment tags. To continue the example, the HTML page received from requesting http://www.xyzcorp.com should contain a string within comment tags somewhere inside the document that matches “loadstorm-90630fc588″. LoadStorm will parse the page returned by your server exactly like a browser will. In that parsing, the verification string must appear somewhere for us to consider it verified.
Example Code with a string in a comment tag:
Once you have the file or text in place, click on the Build… Servers tab, then select the the domain from the table that is your target server. Click the Check/Verify button under the verification section. If you are still having difficulties or need extra assistance, please contact [email protected].
Please only use the domain name when adding a server, and only use the domain name once. Adding a server for http://www.xyz.com works great and verification is successful. However, creating a server for http://www.xyz.com/myfolder will not work and trying to verify it will produce an error.
Wonder why you need to verify servers?
Please see our discussion on why servers need to be verified..
- Why should I ramp up my test load volume over time?
It helps you with performance engineering. We recommend that you do not begin a large test at the maximum number of virtual users. Rather than run a test for 10 minutes beginning with 5,000 and ending with 5,000, we suggest that you perhaps start with 500 users and increase to 5,000 over an hour. This approach has the advantage of allowing your system to get all the parts working properly at smaller load (e.g. caches, threads, database connections) before the heavier volume starts exposing potential bottlenecks in your application.
From what we’ve seen in thousands of load tests with LoadStorm, it is common to see useful patterns in the metrics before the peak traffic is reached, and these metrics point to areas of performance limitations that can be tuned.
Load testing is invariably coupled with performance tuning – an iterative process of test/tune/test/tune. Therefore, we recommend you ramp the volume in order to get more value from the test results.
- Why doesn’t my test start immediately?
LoadStorm needs a little time to prepare the test resources such as the load generation servers. Our system is using each of your scripts and steps to pre-calculate the virtual users’ actions, how many servers will need to be instantiated from the cloud, random think times, etc. This produces essentially a test roadmap that our Scheduler and Summarizer modules use to coordinate all the HTTP traffic and correctly capture the metrics. Depending on the number of concurrent users and the duration of your test, the preparation of a load test can take 2 to 15 minutes.
- How does it generate HTTP traffic against my web application?
LoadStorm is a truly distributed application that leverages the power of Amazon Web Services to scale on demand with processing power and bandwidth as needed to test the largest web projects. As you crank up the testing load from 200 to 150,000 virtual users, LoadStorm is automatically adding machines for you (as many as necessary) from Amazon’s server farm to handle the processing. When your tests are done and the extra machines are not needed, they are turned off to wait for another test.
The normal processing is for a script to be processed as a sequential series of pages that you have defined in the recording. Once a page finishes, the VUser pauses for the specified time (between Min & Max “Think Time” in script), then the next page proceeds. After the last page completes, the VUser context (session, cache, cookies) is cleared and the script starts from the beginning again with a new VUser.
You define the number of concurrent users you want to start with and peak at. As one VUser completes the script as described above, then another VUser is created to replace it – starting at page 1 again.
If you have started at a small number and set the peak number much higher, then LoadStorm will not only retire VUsers that have completed the script, it will also add additional users throughout the test to ramp up the volume.
This is repeated until the test is done as defined by the test duration field (in minutes) when you began the load test. - How Can I Increase the Requests per Second in My Test?
Here are some thoughts on increasing RPS in LoadStorm tests:
- Reducing think time between pages in a script – minimum is 5 second pause, so set both min & max to 5.
- Increase the number of concurrent users.
- Reduce the response time from your server which decreases LoadStorm’s waiting for a response.
RPS will always be impacted by the types of pages/resources your scripts are requesting. The way it works is that LoadStorm cranks up the starting concurrent users shortly after test starts. It then walks each user through the script assigned to it. One step is executed (such as home page), and all the images, JS, CSS, XML are requested separately. When those responses are all returned, then the system pauses a random time between the Min & Max “Think Time”, then it requests the next page. That continues until the end of the script, whereupon that virtual user is retired. After a few seconds, another vuser is created and the process starts again. So, having more users going at the same time, without waiting more than a few hundred ms to get a response, will increase RPS.
- If I stop a test before my max users is reached, are the rest of my users wasted?
Occasionally, a test will be terminated early because the target server fails or the tester stops it. It is not uncommon to have a test automatically stopped by LoadStorm when error rates hit 70% or so. When this happens in a test using virtual users, those users may seem “wasted”.
Customers have asked us why the system deducted 5,000 VUsers from their account when the test only lasted 10 minutes and stopped at 500 users. The reason LoadStorm does not prorate the VUsers to the minute or to the actual peak users is because we incur the full costs of the 5,000 VUser test.
Amazon’s cloud is elastic, and we “rent” the servers needed for your tests. However, there is a minimum of 1 hour increments that Amazon will bill us. For example, if your test needs 20 servers to hit your defined peak users, we buy those 20 servers upfront before your test actually starts. Then if you load test dies 5 minutes into it, we are still charged as if the test used all 20 servers for 1 hour.
Thus, we recommend that you start with smaller tests to verify that you have all of the environmental factors properly configured. Growing the volume in several tests that increase in volume will potentially eliminate the “wasted” feeling of throwing 5,000 users at a server that fails at 500 users.
- Why aren’t the 500 users evenly distributed across all 5 of my servers behind a round robin load balancer?
A round robin load balancer will not work very well for testing your configuration unless you are using a much higher number of virtual users (at least 2000 or so). Each of our simulation servers will resolve the DNS name independently of the others, but each of our servers will simulate hundreds of users. I would expect a peak load of 500 users to get routed to more than one of your servers, but probably not all 5 of them. A peak load of 100 users or less is likely to hit only one of your servers.
- Will the load test show up in the Google Analytics reports? How is testing typically tracked by Google Analytics?
We would not expect LoadStorm test volume to show up in your Google Analytics. LoadStorm does not execute the javascript, and requests outside of your domain are blocked unless those other servers have also been verified. Servers like Google Analytics should be set to “Ignore”. So, the tracking by Analytics, ad servers, or other systems outside of your domain would not work. Some of these kinds of services have restrictions against load testing.
Almost all of our customers don’t want our load testing tool traffic to show up in their marketing statistics (Google Analytics) because it essentially renders their analytics data worthless. Additionally, Google has limits on the amount of hits per period in their free GA account; otherwise you must upgrade to the paying account. There are also legal issues involved whereby Google doesn’t want automated tools like LoadStorm hammering their applications (even GA) – it is a violation of their standard terms and conditions. Thus, when a GA tracking javascript is found in a page that is involved in a LoadStorm test, that server is ignored automatically by our tool.
Most of our clients use a different mechanism for watching the user activity of a load test. There are many tools for monitoring the website, including several excellent options for open source and commercial products such as New Relic, Loggly, and Hyperic. Most of these are very inexpensive or free, up to a certain level. These types of tools provide significant information regarding the user activity as well as important data on your architecture. For example, is your web server memory bound when response times start to increase drastically? Or did your database server hit 99% utilization when throughput drops?
New Relic and Hyperic can monitor server activity from the local server using an agent. Loggly is a tool that has teamed up with New Relic for analyzing server logs which can make your life easier when trying to extrapolate the user activity from your server log.
So to sum it up any javascript that reports traffic to third-party analytic tools will not work with LoadStorm. Instead useful tools for monitoring server usage and analyzing log files of user activity on the server under test (SUT) our encouraged and will work with LoadStorm.
- Can your system run load tests behind my firewall for non-Internet applications?
Unfortunately not. LoadStorm is only for web applications that can be reached from the Amazon cloud. If you have an intranet application, then opening a hole in your firewall will allow LoadStorm to run load tests against it.
- How much does LoadStorm emulate the caching abilities of a real browser client? Will cache be reset across various sessions?
The short answer is: If your browser caches it, LoadStorm will cache it for a virtual user. When a virtual user completes a script, it is retired which causes the session to end.
Our load testing tool is designed to closely emulate the browser with a new, empty cache between sessions. So, generally your static assets will be cached for later pages in a script according to their header values. There are a few exceptions. We do not cache HTML resources since these are usually dynamically generated. Most LoadStorm users expect a subsequent HTML request to get this file again, thus that is how we coded it to work. If the headers are set properly, this will usually happen anyhow.
- What happens to a virtual user when something fails or times out in a scenario?
The vuser continues to move to the next step in its scenario. For instance, if a JavaScript file fails to download during step 3:- That error will be logged in LoadStorm for reporting
- That request will be terminated
- All other requests for step 3 will be attempted as usual
- The vuser will begin step 4 as defined in the scenario
If any particular resource times out within a step, some of the step processing may not finish. This depends on what kind of resource fails and at what point in the step. If an image fails, this is relatively harmless. The images following will not be requested, but everything else will continue normally. If the html fails with a timeout, then the step will stop and no images or other resources will be requested. Images, javascript and css all depend on the html, so none will be requested for that step unless the html is received without error.
The step following a step with an error will work normally to the extent that it can. As with a browser, you cannot click on a link or submit a form if you do not currently have a valid page available with which to work. You can always type in a new url and go from there. Likewise, in LoadStorm if the next step is an open step, then previous steps will have no effect except for possibly login or cookie state. If the next step is a click or form, then the previous html must have been received for the step to function. If the step cannot work because the previous html timed out or otherwise failed, then that step will show no requests or errors and the next step will be attempted after the expected pause.
There is currently only one situation where a scenario will restart before all of its steps have run. This is a step timeout where a step starts but does not finish before 5 minutes have passed. A step includes getting all resources within the step such as html, javascript, images, css, etc. So, it is not inconceivable that this timeout occurs if there are 50, 100 or more resources in a step, but it is long enough that it is pretty rare.
- What happens to virtual users as test scripts are running?
LoadStorm creates the starting concurrent users within seconds of the test starting. For example, if you set the test to start at 1,000 users then you will have requests coming from those 1,000 users in the first 30 seconds. It then processes each virtual user through the script assigned to it. The first page, for example your home page, is execute first. The HTML is sent from your server and LoadStorm receives it. All other resources recorded (images, Javascript, CSS, and XML) are requested separately and sequentially.
When all responses in a page are returned, the system pauses a random time between Min & Max Think Time pause (you can edit this), then it requests the next page. That continues until the virtual user reaches the end of the script, whereupon that VUser is “retired”. After about 30 seconds, another VUser is created and the process starts again. The cache for the new virtual user starts empty and does not remember cookies or static resources for the previous VUser.