- How can I use analytics to estimate the number of virtual users to ramp up to during my load test?
That’s a good question. Nowadays tools like Google Analytics can offer real time monitoring which will give you the best insight into your peak levels of traffic and how that traffic may spike for a few minutes. If you have this information it already provides a good estimate for many peak users you should test for in LoadStorm.
Most commonly though Google Analytics will have data on the number of visits per hour to your site as well as the average duration of those visits. This makes it a bit harder to estimate what volume of users you should test for. With this information we can estimate the average concurrent visitors that were on your site using the following equation:
U = V / (60/D)
Where: U is the number of load test virtual users (that’s what we are trying to figure out) V is the average number of visitors per hour D is the average duration of a visitor
60 is the number of minutes in an hour
Let’s state this formula again in English like a math word problem:
Load Test Virtual Users is equal to the Average Visitors per Hour divided by the User Turnover Rate per Hour.
Ideally you should test for an amount that is a bit greater than this estimate just to make sure you have a cushion for unusual spikes in traffic. For a more lengthy explanation please visit our blog post about virtual user calculations.
- How do I build a script and schedule a test run?
Excellent question. Click here for step-by-step instructions to begin using LoadStorm.
- Why should I ramp up my test load volume over time?
It helps with performance engineering. We recommend to not begin a large test at the maximum number of virtual users. Rather than run a test for 10 minutes beginning with 5,000 and ending with 5,000, we suggest to start smaller, for example, with 500 users and increase to 5,000 over an hour. This approach has the advantage of allowing your system to get all the parts working properly at smaller load (e.g. caches, threads, database connections) before the heavier volume starts exposing potential bottlenecks in your application. Also if the target server begins to fail during the test this provides more granularity into the number of concurrent users where failure begins. If the test starts at 5,000 and fails all that can be learned from the results is that it fails somewhere between 1 and 5,000 which isn’t very useful.
From what we’ve seen in thousands of load tests with LoadStorm, it is common to see useful patterns in the metrics before the peak traffic is reached, and these metrics point to areas of performance limitations that can be tuned.
Load testing is invariably coupled with performance tuning – an iterative process of test/tune/test/tune. Therefore, we recommend you ramp the volume in order to get more value from the test results.
- Why doesn’t my test start immediately?
LoadStorm needs a little time to prepare the test resources such as the load generation servers. Our system is using each selected script(s) to pre-calculate the virtual users’ actions, how many servers will need to be instantiated from the cloud, random think times, etc. This produces essentially a test roadmap that our Scheduler and Summarizer modules use to coordinate all the HTTP traffic and correctly capture the metrics. Depending on the number of concurrent users and the duration of your test, the preparation of a load test can take 2 to 15 minutes.
- How does it generate HTTP traffic against my web application?
LoadStorm is a truly distributed application that leverages the power of Amazon Web Services to scale on demand with processing power and bandwidth as needed to test the largest web projects. As you crank up the testing load from 200 to 1,000,000 virtual users, LoadStorm is automatically adding machines for you (as many as necessary) from Amazon’s data centers to handle the processing. When your tests are done and the extra machines are not needed, they are turned off to wait for another test.
A script is processed as a sequential series of pages that you have defined in the recording. Once a page finishes, the VUser pauses for the specified time (between Min & Max “Think Time” in the script), then the next page proceeds. After the last page completes, the VUser context (session, cache, cookies) is cleared and the script starts from the beginning again with the same VUser. This same VUser will use different data upon script repeat if a CSV of User Data or dynamic response text has been parameterized within the script. As the concurrent users scale up they each begin the script for the first time, and also repeat the script after each completion.
You define the number of concurrent users you want at the start and peak of the test. You can also define a ramp down period to see how well your system recovers as the traffic subsides.
Scripts are repeated until the test is done as defined by the test duration field (in minutes) from when you began the load test.
- How Can I Increase the Requests per Second in My Test?
Here are some thoughts on increasing RPS in LoadStorm tests:
- Reducing the think time added after each page in a script; the minimum pause is 5 seconds, so set both min & max to 5 seconds.
- Increasing the number of concurrent users.
- Reducing the response time from your server which decreases the time LoadStorm has to wait for a request to complete.
- Compressing the size of the response is one way to decrease the time it takes for the request to complete as well as save on throughput.
RPS will always be impacted by the types of pages/resources your scripts are requesting. The way it works is that LoadStorm cranks up the starting concurrent users shortly after test starts. It then walks each user through the script assigned to it. The GET requests will be made in asynchronous bursts of up to 6 at a time such as HTML, images, JS, CSS, and XML. Each POST request will be made on its own to ensure that any dynamic response text from earlier requests was available for to be passed into the POST data. This also ensures that any response data from the POST is available for subsequent requests. When those responses are all returned, then the system pauses a random time between the Min & Max “Think Time”, then it requests the next page. This process continues until the end of the script. So, having more users going at the same time, without waiting more than a few hundred ms to get a response, will increase RPS.
- If I stop a test before reaching the peak users, are the rest of my users wasted?
You can terminate a test early because the target server fails. When this happens in a test using virtual users, those users may seem “wasted”.
Customers have asked us why the system deducted 5,000 VUsers from their account when the test only lasted 10 minutes and stopped at 500 users. The reason LoadStorm does not prorate the VUsers to the minute or to the actual peak users is because we incur the full costs of the 5,000 VUser test.
Amazon’s cloud is elastic, and we “rent” the servers needed for your tests. However, there is a minimum of 1 hour increments that Amazon will bill us. For example, if your test needs 20 servers to hit your defined peak users, we buy those 20 servers upfront before your test actually starts. Then if your load test dies 5 minutes into it, we are still charged as if the test used all 20 servers for 1 hour.
Thus, we recommend that you start with smaller tests to verify that you have all of the environmental factors properly configured. Growing the volume in several tests that increase in volume will potentially eliminate the “wasted” feeling of throwing 5,000 users at a server that fails at 500 users.
- How do I stop a test that has killed my server?
You can press the “Stop Execution” button during a load test. To do this:
- click ANALYZE in the left navigation
- double-click the test run that is currently in progress
- near the top-right in the filters area is the “Stop Execution” button
- Why aren’t the 500 users evenly distributed across all 5 of my servers behind a round robin load balancer?
A round robin load balancer will not work very well for testing your configuration unless you are using a much higher number of virtual users (at least 1000 or more). Each of our simulation servers will resolve the DNS name independently of the others, but each of our servers will simulate hundreds of users. I would expect a peak load of 500 users to get routed to more than one of your servers, but probably not all 5 of them. A peak load of 200 users or less is likely to hit only one of your servers.
- Will the load test show up in the Google Analytics reports? How is testing typically tracked by Google Analytics?
We would not expect LoadStorm test volume to show up in your Google Analytics. LoadStorm does not execute the javascript, and requests outside of your domain are blocked unless those other servers have also been verified. Servers like Google Analytics should be set to “Ignore”. So, the tracking by Analytics, ad servers, or other systems outside of your domain would not work. Some of these kinds of services have restrictions against load testing.
Almost all of our customers don’t want our load testing tool traffic to show up in their marketing statistics (Google Analytics) because it essentially renders their analytics data worthless. Additionally, Google has limits on the amount of hits per period in their free GA account; otherwise you must upgrade to the paying account. There are also legal issues involved whereby Google doesn’t want automated tools like LoadStorm hammering their applications (even GA) – it is a violation of their standard terms and conditions. Thus, when a GA tracking javascript is found in a page that is involved in a LoadStorm test, that server is ignored automatically by our tool.
Most of our clients use a different mechanism for watching the user activity of a load test. There are many tools for monitoring the website, including several excellent options for open source and commercial products such as New Relic, Loggly, and Hyperic. Most of these are very inexpensive or free, up to a certain level. These types of tools provide significant information regarding the user activity as well as important data on your architecture. For example, is your web server memory bound when response times start to increase drastically? Or did your database server hit 99% utilization when throughput drops?
New Relic and Hyperic can monitor server activity from the local server using an agent. Loggly is a tool that has teamed up with New Relic for analyzing server logs which can make your life easier when trying to extrapolate the user activity from your server log.
So to sum it up any javascript that reports traffic to third-party analytic tools will not work with LoadStorm. Instead useful tools for monitoring server usage and analyzing log files of user activity on the server under test (SUT) our encouraged and will work with LoadStorm.
- Can your system run load tests behind my firewall for non-Internet applications?
Unfortunately not. LoadStorm is only for web applications that can be reached from the Amazon cloud. If you have an intranet application, then opening a hole in your firewall and providing a URL or public IP address that can reach the intranet application will allow LoadStorm to run load tests against it.
- How much does LoadStorm emulate the caching abilities of a real browser client? Will cache be reset across various sessions?
The short answer is: If your browser caches it, LoadStorm will cache it for a virtual user. When a virtual user completes a script it closes the session. Then it clears all cookies and cache before repeating the script.
Our load testing tool is designed to closely mimic the browser, but it offers three settings for caching behavior. There are a few exceptions. We do not cache HTML resources since these are often dynamically generated.
