According to the company blog, BrowserMob was released to the public Nov 25, 2008. This is a new load testing product. The advertised benefits of this product over its competitors are a cheaper cost structure, no need for the installation of cumbersome tools, no need for your own testing hardware, and unrestrained scalability. The primary promoter of this product is Patrick Lightbody. Every entry in the blog was written by him and is the primary driving force behind this product. He is involved in the development of Selenium IDE and RC. Therefore, the software behind BrowserMob is probably driven by the Selenium RC core. His full history can be reviewed in his LinkedIn profile. The product website is clean and informative. The provided documentation will quickly allow your tests to be underway. However, the blog and faq sections did not provide particularly useful information. It was targeted toward users unaware of load testing terminologies and practices.
The Product
http://browsermob.com/load-testing
This product is heavily dependent upon the open source effort of Selenium. All shortcomings of the Selenium IDE are inherent in this product due to the fact that all scripts are written using the IDE. However, this product has added some additional functionality to the IDE creation. The ability to use individual usernames as the script runs allows BrowserMob to both login to selected services and avoid errors caused by virtual users sharing usernames.
The virtual users are hosted from the Amazon EC2. Screenshots of test failures are served from the Amazon S3. The test of BrowserMob functionality against a target server within the EC2 eliminates most bandwidth restrictions.
A trial subscription can be obtained quickly using only an email address. The trial consists of 50 credits. One virtual user for up to an hour is one credit. Up to an hour of load test reporting is 5 credits. The results can be stored on the BrowserMob site for later access, but it will cost the customer. The test results can be downloaded from the site in the form of a SQL file. Additional credits can be purchased at any time to allow additional use of the BrowserMob capabilities. Credits can be purchased in a variety of size packages.
The Test
The Script
| BrowserMobTestCase4 | ||
| open | http://cia-factbook.loadstorm.net/ | |
| waitForPageToLoad | 5000 | |
| setSpeed | 4000 | |
| click | link=Appendixes | |
| waitForPageToLoad | 5000 | |
| click | link=D | |
| click | //a[contains(@href, ‘appendix-c.html’)] | |
| waitForPageToLoad | 5000 | |
| click | link=Cross-Reference List of Hydrographic Data Codes | |
| waitForPageToLoad | 5000 | |
| select | CountrySelect | label=Japan |
| click | //tr[21]/td[2]/a[3]/img | |
| clickAndWait | wfbTopBanner | |
| click | link=History of The World Factbook | |
| goBack | ||
| clickAndWait | link=Reference Maps | |
| click | //img[@alt=’Europe Map’] | |
| goBack | ||
| click | //img[@alt=’North America Map’] | |
| goBack | ||
| click | //img[@alt=’Southeast Asia Map’] | |
| goBack | ||
| click | //img[@alt=’Physical Map of the World Map’] | |
| goBack | ||
| click | //img[@alt=’Political Map of the World Map’] | |
| goBack | ||
| click | //img[@alt=’United States Map’] | |
| goBack | ||
| click | //tr[3]/td[5]/span/a[1] | |
| click | wfbTopBanner | |
| click | link=Guide to Rank Order Pages | |
| click | link=Population | |
| goBack | ||
| click | link=Birth rate | |
| goBack | ||
| click | link=Death rate | |
| goBack | ||
| click | link=Internet hosts | |
| goBack | ||
| click | link=Area – total | |
| goBack | ||
| goBack | ||
| click | link=Flags of the World | |
| click | //img[@alt=’Flag of Australia’] | |
| goBack | ||
| goBack | ||
| click | link=Gallery of Covers | |
| click | //img[@alt=’1993 Front Cover’] | |
| goBack | ||
| goBack | ||
| click | link=Notes and Definitions |
In order to test these new capabilities, I created a test script to be used under the trial subscription. The target site is http://cia-factbook.loadstorm.net/index.html. The target site consists of static pages served from an Apache server. The test case consists of browsing around text and image content. The browser will go down into some branches and use the back operator to return to the landing page.
Test cases must be built using the Selenium IDE. Once the script is completed and a functional test is run in the IDE, the load test can be configured.
The test must be configured to run 15 minutes in advance.
The test has a 5 minute ramp up period in case the server cannot handle the maximum user load of 50 virtual users. Once the ramp up is completed, a constant load of 50 virtual users will run the script against the target for a period of 25 minutes. If the server is near load, then the server may see a slow decline in performance as requests begin to build up. Results should indicate the slow decline in performance of the target server.
At this point, scripts are assigned to the load test. Scripts are assigned a percentage of the total running virtual users. Virtual users will repeat their assigned script until the end of the given duration. This does not allow the ability to change the scripts or their allocation percentages later in the test duration.
The Results
Both the anticipated tests and the currently running tests are viewable in the home dashboard. These tests can be deleted, stopped, or paused at any time.
The test info summary is shown above. The number of transactions reveals a count of the number of times a virtual user either succeeded or failed a test. The number of failures is the count of transactions which were determined to be failures. In the event of a failure, a screenshot of the browser at the time of failure should be recorded. The documentation indicates that the first 50 failures will have their screenshots recorded. I was not able to determine the reason why this test failed to record any screenshots. The sheer number of failures indicates a serious error. The number of objects are raw retrievals of data using requests.
The green line indicates the target number of users. The blue line indicates the actual users running at the time.
This chart reveals the throughput of the system as it makes requests upon the target. This can be broken down by the various test cases entered.
This chart reveals the response time of the test. The test can be broken down into the steps within the scripts. Unfortunately, the errors within the test prevented any steps beyond the first to be revealed.
This chart reveals the transactions as broken down by success or failure at step number. In this graph, it is shown that the test maintained large amounts of errors throughout the entire test.
Now that the test is complete, the results can be examined in further detail to determine why the test may have failed.
The BrowserMob uses a MySQL database to store test results. This message indicates the cause of the many errors occurring within the load test. Initial investigation showed that the favicon.ico file did not exist on the target server which was causing a failure. However, subsequent tests continued to fail with this error even when the file was present.
The Review
I would have preferred to review this product with a working demonstration. In the end, I could not determine the cause of the error. The provided documentation required only that the script succeed within the Selenium IDE. Additionally, the failed tests did not give the screenshots of the browser at the time of failure. The documentation specifically mentions that the first 50 errors should have included screenshots.
The four graphs and their layout are adequate to convey the overall status of the test. I would have liked to see more macro data. For example, a breakdown of what types of failures were observed would be helpful. The important error types consist of 404 responses, missing content in pages, and timeouts. The ability to delve into each individual request is useful for occasional errors. The individual details of successful steps are not required.
The largest extent of my time was spent correcting problems with the Selenium recording. Specifically, Selenium would attempt to click on a page that was not fully loaded. It was not always appropriate to wait for the entire page to load since this is not always typical user behavior. I had to add some short pauses to correct this behavior. Finally, Selenium did not record go back actions and I was forced to add them manually. These behaviors are indirectly part of the BrowserMob experience and detract from the overall user friendliness.
The test scripts are repeated until the end of the time period. An alternate option of specifying the number of times to run the scripts would be useful. This would prevent the premature ending of running scripts. The documentation mentions that the premature death of scripts can cause the error code -999.
* Update 1/11/2009 – Patrick send some info to us, along with the fixed test script (available for download).