- How can I use analytics to estimate the number of virtual users to ramp up to during my load test?
That’s a good question. Nowadays tools like Google Analytics can offer real time monitoring which will give you the best insight into your peak levels of traffic and how that traffic may spike for a few minutes. If you have this information it already provides a good estimate for many peak users you should test for in LoadStorm.
Most commonly though Google Analytics will have data on the number of visits per hour to your site as well as the average duration of those visits. This makes it a bit harder to estimate what volume of users you should test for. With this information we can estimate the average concurrent visitors that were on your site using the following equation:
U = V / (60/D)
Where: U is the number of load test virtual users (that’s what we are trying to figure out) V is the average number of visitors per hour D is the average duration of a visitor
60 is the number of minutes in an hour
Let’s state this formula again in English like a math word problem:
Load Test Virtual Users is equal to the Average Visitors per Hour divided by the User Turnover Rate per Hour.
Ideally you should test for an amount that is a bit greater than this estimate just to make sure you have a cushion for unusual spikes in traffic. For a more lengthy explanation please visit our blog post about virtual user calculations.
- Why did my QuickStorm fail?
Our QuickStorm feature is meant to help give a quick demonstration of what our reporting will look like during a test run. To make this process as smooth as possible we remove the task of having you make a browser recording, saving it as a HTTP Archive (HAR) file, uploading it to LoadStorm to be converted into a script and scheduling a test run. Instead we take a URL you provide and feed it into an automated process that handles these steps for you. Anytime you try to feed in complex elements into an automated process there are bound to be some issues that may prevent the process from completing.
Sometimes these are simple mistakes:
- typo mistakes in the URL
- requesting URLs using HTTP when they need HTTPS
- private URLs
- local IP addresses
- URLs that need basic authentication credentials
Other times there are more complex issues with the automated tool handling the page given to it. Usually this is due to how it fails to parse something in the page:
- handling WebSockets
- malformed html
- javascript errors
- streaming audio/video
- failing to make a connection for a requested resource
Many of these are not an issue with real browser recordings. So if you encounter a problem please use the LiveChat button at the top of the QuickStorm test and ask for assistance, or email [email protected] to request a demo. Otherwise we encourage you to check out this guide for getting started with LoadStorm to create a real browser recording and run your first load test instead of a QuickStorm.
- How do I build a script and schedule a test run?
Excellent question. Click here for step-by-step instructions to begin using LoadStorm.
- What does LoadStorm do?
LoadStorm™ is a web-based load testing tool for simulating what users do with a web site or web application. You use it to build tests that send requests to your server in the same way that a user’s browser sends requests to your server. But these tests are executed by our automated systems rather than by a user, so they can be done repeatedly and in large numbers simultaneously. They can also be built using our tool in such a way as to simulate a large number of different users with different tasks to perform. - How much does it cost?
It depends on how many virtual users you need for testing, and it depends on if you would like to use our consulting services.Creating an account is free, but a free account is limited to 50 virtual users to try out our product. By default, we offer four pricing plans. Each plan scales up in cost, but offers more benefits such as bundled consulting hours.
For more details, please see Load Testing Cost.
- Do I need to manage my servers? How do I do that?
Depending on the test requirements, third-party servers should be set to “Ignored” or “Archived”. Requests that would go to ignored or archived servers will be skipped instead. Skipped requests prevent unnecessary traffic to advertisements, analytics, or other third-party services. You may encounter a server already set to ignored as a part of our blacklist. Please contact [email protected] about enabling a blacklisted server for your testing needs.
Ignore a Server
Please note that by ignoring a server, any requests normally sent to that server will not be sent.
- Click on the Build menu, then on the Server tab on the top.
- Double-click the server you want to ignore.
- Click the Ignore button. No requests will be sent to this server.
Archive a Server
Archiving a server works the same as ignoring a server, but it’s a separate category. This way you can separate the servers you may never need to test from servers that you may not want to test at the moment. To reiterate, archiving a server means it is ignored. Any requests normally sent to that server will not be sent.
- Click on the Build menu, then on the Server tab on the top.
- Click the server you want to archive to select it.
- Click the Archive button. No requests will be sent to this server.
- How should I set up scripts to simulate real traffic?
Most people create several scripts to accurately reflect different types of users. Scripts execute simultaneously in order to hit your site with the correct mix of traffic. You control the weighting of each script by giving it a number relative to the others. For example, if you have 2 scripts and you give one a weighting of 4 and the other a weighting of 1, then the first will generate 80% of the traffic and the second 20%. This 80/20 could be realistic for a retail e-commerce site where most of the people (80%) are browsing the product catalog, while a minority of people (20%) are going through a buying experience. You can of course throw in a low-weight scenario to represent some internal employees adding products or issuing refunds.
Another way of entering the weight of the traffic is to just enter the exact percentage you would like each script to carry. For the example above you would simply enter 80 and 20. The only thing you have to be very careful about is that the percentages add up to 100. Otherwise, it will work like the above description, which is different.
There is no limit to the number of scripts you can create, and you can use many scripts in a test. We encourage you to build as many as necessary to accurately reflect the real traffic patterns on your site. That will give you the most meaningful metrics for tuning your applications.
We also simulate the user “think time” with random pauses after each page. Real users click on something, then read the page or fill in a form, and then click on something else. Thus, you don’t want to have a simulated virtual user clicking on something every .5 seconds – it would not give you meaningful results if you want to see how your site holds up in the real world.
Also see: Virtual User Calculations about calculating the amount of virtual users needed for a realistic test.
- Why should I ramp up my test load volume over time?
It helps with performance engineering. We recommend to not begin a large test at the maximum number of virtual users. Rather than run a test for 10 minutes beginning with 5,000 and ending with 5,000, we suggest to start smaller, for example, with 500 users and increase to 5,000 over an hour. This approach has the advantage of allowing your system to get all the parts working properly at smaller load (e.g. caches, threads, database connections) before the heavier volume starts exposing potential bottlenecks in your application. Also if the target server begins to fail during the test this provides more granularity into the number of concurrent users where failure begins. If the test starts at 5,000 and fails all that can be learned from the results is that it fails somewhere between 1 and 5,000 which isn’t very useful.
From what we’ve seen in thousands of load tests with LoadStorm, it is common to see useful patterns in the metrics before the peak traffic is reached, and these metrics point to areas of performance limitations that can be tuned.
Load testing is invariably coupled with performance tuning – an iterative process of test/tune/test/tune. Therefore, we recommend you ramp the volume in order to get more value from the test results.
- Why doesn’t my test start immediately?
LoadStorm needs a little time to prepare the test resources such as the load generation servers. Our system is using each selected script(s) to pre-calculate the virtual users’ actions, how many servers will need to be instantiated from the cloud, random think times, etc. This produces essentially a test roadmap that our Scheduler and Summarizer modules use to coordinate all the HTTP traffic and correctly capture the metrics. Depending on the number of concurrent users and the duration of your test, the preparation of a load test can take 2 to 15 minutes.
- What are the IP address ranges used by LoadStorm? We need to white list them.
- If I stop a test before reaching the peak users, are the rest of my users wasted?
You can terminate a test early because the target server fails. When this happens in a test using virtual users, those users may seem “wasted”.
Customers have asked us why the system deducted 5,000 VUsers from their account when the test only lasted 10 minutes and stopped at 500 users. The reason LoadStorm does not prorate the VUsers to the minute or to the actual peak users is because we incur the full costs of the 5,000 VUser test.
Amazon’s cloud is elastic, and we “rent” the servers needed for your tests. However, there is a minimum of 1 hour increments that Amazon will bill us. For example, if your test needs 20 servers to hit your defined peak users, we buy those 20 servers upfront before your test actually starts. Then if your load test dies 5 minutes into it, we are still charged as if the test used all 20 servers for 1 hour.
Thus, we recommend that you start with smaller tests to verify that you have all of the environmental factors properly configured. Growing the volume in several tests that increase in volume will potentially eliminate the “wasted” feeling of throwing 5,000 users at a server that fails at 500 users.
- Why not let our users test the system’s capacity?
For some projects, this is an acceptable approach. For some internal applications, apps that are very simple will realistically expect light traffic. The best plan may be to get it done as quickly as possible. You have to ask yourself what is the risk of the application not working at some point. For many businesses, getting a large influx of new users is the peak of success, and that success would be shattered by having the new users unable to use the system, unable to buy your product and unhappy with your company.