Before performing any sort of test, it is best to plan the expected behavior of the test. In this particular instance, the developer is performing a load test of static pages on the Google App Engine. The App Engine is a specialized Python API tied into a limited computing cloud. The developer is this test will be using the EC2 cloud provided by Amazon Web Services to provide the ramping behavior of multiple virtual users. The five stages of the test are setup, functional test, single node load test, multiple node load test, and interpretation of the resulting data. The following provides the necessary steps to setup static page serving on the GAE, as well as, a look into what behavior I would expect from a load test between these two clouds.
Metrics
The files used on the GAE as well as the data received from the LoadStorm and the GAE DashBoard will be made available.
Setup
Retrieve the necessary static files for a target server to fill the quota of space. Configure the application to properly serve the files given the appropriate URLs.
Functional Test
One virtual user should be able to go through the site without encountering any problems.
Load Test
The load test should consist of two parts.
First, a ramping test to measure the load that a single LoadGenerator can handle. This will give performance metrics for future reference in determining the max load that a single LoadGenerator server can handle.
Second, a ramping test of multiple machines in order to test the ability of the system to handle multiple systems under load as well as to ascertain any possible load upon the Google App Engine.
Interpret
Create conclusions as to any successes and/or failures. These must include any expectations that were not met or exceeded. As well as, any recommendations for future tests.
Analysis
The files downloaded from the Apache site are the standard download package of log4j. These files included a local copy of the log4j site as well as the compiled API documentation. This package included 753 files of a variety of types including html, css, and images which are to be used.
In order for the static files to served from the GAE, modifications had to be made to the “app.yaml” file. This file indicates in what manner the files should be served given a particular URL request is passed to the application. The proper directions where placed as to correctly serve images and css files in support of the html pages present.
http://loadstorm.com/files/test_files.zip_.txt
A txt extension had to added to the end of the file in order to upload. It can safely be removed.
Expected Behavior
The expected behaviour of the App Engine over a single day’s period can be reasonably predicted when given a heavy traffic load. The anticipated behaviour of a single LoadGenerator operating on a typical high-traffic server would look something like this:
This is the slow degradation of performance of a LoadGenerator as it can no longer handle the traffic. In an ideal load test, all LoadGenerators will not be loaded to such a degree. This is to prevent skewing of the data by LoadGenerator lag. The ideal measurement would remove the LoadGenerator lag from the results of the target.
“The rate at which your application can consume resources will instantly be increased and initially all of the requests for your application will be served as the traffic starts to increase. This “spike rate” will then gradually decrease in order to ensure your app will continue to serve throughout the day while remaining under the 24-hour quota limits”.
The is the expected graph of the GAE assuming that resources are not exhausted almost immediately. GAE will allow the traffic ramp-up caused by the LoadGenerators. Once it peaks, the GAE will begin restricting the bandwidth in order to preserve the remaining day’s quota.
Quota Limit Emails per Day 2,000 Bandwidth In per Day* 10,000 MB Bandwidth Out per Day* 10,000 MB HTTPS Bandwidth In per Day 2,000 MB HTTPS Bandwidth Out per Day 2,000 MB CPU Megacycles per Day 200,000,000 Total HTTP Requests per Day 650,000 HTTPS Requests per Day 130,000 Datastore API Calls per Day 2,500,000
URLFetch API Calls per Day 160,000
Once the quota for a given day or perhaps even a smaller increment of time is used, the application will begin to server “403” errors to any users. Once the quota period previously discussed has ended, a new quota period will begin with the same limitations and zero percent usage.
Quotas
http://code.google.com/appengine/articles/quotas.html
In order to monitor the expected activity from the perspective of the Google App Engine, the developer has access to the Dashboard. This feature provided by GAE provides near real-time monitoring of the activity of the application.
The ramped test is constricted by 24 hr periods and a number of quotas within that period. These conditions are suitable for a single LoadGenerator performance test. It is not certain how many servers can be added to conduct a true load test without adding skew from quota restrictions. Once a single load test is conducted, general knowledge of the limitations of a single LoadGenerator can be surmised. In particular, how many virtual users can a LoadGenerator handle and how much of the maximum quota was utilized for the test. The data for a single server should be identical to any other server with the same specifications. It should provide the answer to the scale of the load test available.
If the limitations of the current quota prove overly restrictive, then a request to GAE may be submitted to increase the maximum Quota.