WebPageTest (WPT) is a free, open source tool that is used to evaluate web page performance. The great thing about WPT is that it can be utilized by people with all levels of expertise, and has many options for advanced settings and customizations. The public WPT instances cover many scenarios, but private instances can be useful, especially when testing sites that aren’t yet publicly accessible.
We decided to install our own private instance of WPT using an Amazon Micro EC2. However, as the WebPerfLab, we couldn’t just leave it at that. Come on, you should know us by now! We had to know; how would this affect our performance test results?
To determine this we ran similar tests using the private and public instances to compare results. The target pages and the browsers used were the same, and the tests were run from similar locations. WPT was set to Dulles, VA, and our EC2 is in Amazon’s US East datacenter located in Northern Virginia.
As you can see the private instance was loading our target pages one second slower on average than the public tests. Our initial guess was that the difference in load time could probably be attributed to a difference in bandwidth, so we decided to test that theory.
We ran several speed tests from our micro EC2 using SpeedTest.net and compared the results. It’s clear from our results that even though our EC2 experienced large fluctuations in bandwidth, the average speed was still much faster than what we needed for the performance tests (5 Mb/s download and 1 Mb/s upload). A difference in bandwidth wasn’t the answer.
After doing some more digging, we found some interesting information to corroborate the differing results which we were experiencing. We found an old article that gives similar test results showing a difference between a micro EC2 and a small EC2 as the host server. Patrick Meenan, the head honcho of WebPageTest, was quoted in the article stating that a micro EC2 is simply not good for performance testing. With the limited CPU and memory, the results fluctuate too wildly.
This leaves us with an important lesson: Taking a single measurement point and using it as a representative for the performance of a site is not enough. Even using a platform that is back-end consistent can yield significant variations. Pairing LoadStorm test results with WebPageTest results can give much more meaningful insight and perspective on how a site is really performing.