LoadStorm™

Happy New Year! I mean that literally...it is a happy new year for web performance geeks like you and me.

Did you ever read The Hobbit? There was a clever exchange between Gandalf and Bilbo where the meaning of "Good Morning" was discussed. Are you wishing me to have a good morning or are you saying it's a good morning whether I want it or not?

Well, 2012 is going to be a great year for the web performance and load testing industry. I wish that you have a happy new year, AND it will be a happy web performance new year whether you want it or not.

Load Testing for Holiday Season was Awesome

LoadStorm begins 2012 with an upbeat outlook because the last few months of 2011 showed over a 400% increase in load testing volume.

The actual number of load tests being executed by our customers were maybe only double the number in 2010, but the scale of tests was much bigger. It is also interesting to note that many more online retailers were running tests of 25,000+ concurrent users. We had many calls with traditional brick and mortar companies that were putting significant investment into their web store capabilities - including speed and scalability.

Anecdotally, we can share with you that it was a good investment because some of those customers told us that their online sales had risen as much as tenfold (10x) over the previous year! That's great news for all of us web performance engineers.

Not only does it seem clear to us that web stores are getting high priority for e-commerce, but it is also clear that web performance has gotten much more attention from the C-suite. Several of our customers mentioned their load testing projects were being driven from executives worried about site crashing under heavy traffic. That's outstanding! Finally, the stories of Web site performance failures is getting the attention it deserves. I guess it was tough to ignore all the headlines blasting companies like Target when their site crashed in 2011.

Operations and marketing leaders are starting to understand the correlation between web performance tuning and profitability. Web sites are not just online brochures, nor are they just a secondary revenue channel. The message that is coming through loud and clear is that faster sites make more money.

In more than one project-related conference call with companies running load tests of 50,000-100,000 concurrent users, there was a VP of Marketing being very active in driving her/his team regarding the results he/she was expecting. It was refreshing (somewhat shocking) to hear a 60 year old traditional advertising agency veteran telling everyone on the call that "sub-second response time is imperative to success!" I loved it. The web coders...not so much because they had lots of optimization ahead of them.

So far in our series on Web Performance Optimization, we've focused on how to reduce the number of requests between client and server through caching, and how to make requests more efficient by managing server resources. Another strategy in Web optimization is intelligent distribution of resources across the Internet, which can greatly reduce request latency by locating redundant copies of Web content on multiple servers spread across the Internet. In this installment of our series, we focus on content delivery networks (CDN), a technology that increases throughput by bringing content closer to the people requesting it.
 

What is a CDN?

In the simplest Web site configuration, a single Web server services requests from multiple clients. While this is often good enough for the simplest, lowest traffic Web sites, complex Web sites that need to scale to thousands or millions of visitors require more processing power. This is why many sites have resorted to using Web server farms, which are clusters of multiple Web servers offering redundant copies of a site's content. Web farms use load balancing software to monitor the amount of load on any one server. They can also use this information to route requests to the server with the least load at a given point in time.

A CDN is a type of Web farm or server cluster, except that instead of using a single farm or cluster, the servers are spread out over the Internet in multiple geographical locations. These are called edge servers, because they are located at the extremes, or edges, of the Internet, instead of all being located off of a central Internet backbone link. The goal of a CDN is to decrease the time it takes to deliver content to a specific user based on that user's location.

Let's say, for example, that a company based in New York receives a request for Web content from a user in Seattle, WA. In a traditional setup, the Seattle computer's request would have to find the most efficient route on the Internet to New York, usually via a busy backbone link. In a CDN configuration, the CDN could tell the Seattle client that its nearest edge server is on a subnet in Portland, Oregon. By obtaining the content from a server closer on the network, the client greatly reduces request latency.
 


In my last article, I wrote about the paradigm shift in web application architecture and why performance testers have to re-think their strategy around testing Rich Internet Applications (RIA) for performance. Web application development processes and user expectations continue to grow by leaps and bounds. Sadly, the techniques and approaches employed to test those applications have not kept up with the same growth rate. But the good news is that newer tools are coming up and methodologies are being defined to close in on that gap. Hence, it is essential that performance testers make use of them at every phase of the performance testing lifecycle.
 
Early on in the performance testing lifecycle, testers gather requirements and collecting application usage statistics is typically one of the primary tasks. In this article, I will explain how “Web Analytics tools” can be a great source of information to gather historical data about the application usage and user behavior.


 

Traditional Web Server Log Approach


Traditionally, performance testers have been relying on the web server log files to collect historical application usage data. A web server log was and still is a great source of information. They contain enormous data on web usage activity and server errors. Downloading log files from the web server and running report generation tools will help testers get meaningful info out of them. However, web server logs have their limitations. For example,

• Usage data contained in the web server logs do not include most “page re-visits” due to browser caching. For e.g. If a user re-visits a page, no request is received by the web server as the page is retrieved from browser cache. 
• 
While data contained in the Web server logs can provide insights into system behavior, it does not help much in understanding “user/human behavior”. 
• 
Web server logs do not provide user’s geographical info, the browser they used and the device/platform they accessed the application from. All of which are vital metrics to understand user behavior on the application. 

While Web Server log files are still a great way to measure user statistics, new ways to measure web traffic have propped up that provide information from a user-perspective rather than a system perspective. A large number of organizations are implementing what is called “Web Analytics Tools” as part of their Web application infrastructure. For example: Industry reports suggest that Google analytics, a leading Web Analytics tool provider is used on 57% of the top 10,000 websites.

This article will explain the risks associated with using a scaled (aka downsized) environment for Performance Testing. I’ve been a little off topic lately and I thought I would jump back into the realms of Performance Testing. I thought I would attempt to answer one of the most complicated questions I’m faced with when Load Testing. The question is this “If we half the size of the performance/load testing environment can’t we just multiple the figures up?” This is a straightforward question and the answer is simple – ‘NO’. But justifying the answer and explaining in simple terms is more difficult. Particularly to PM’s and people not directly attached to the technology. So I’m going to attempt to answer in simple terms why scaled load testing environments tend not to work and highlight the risks to be considered when using them. Point people at this article if you struggle to answer this question – and let me know what they think.

First lets take an object: A square – if we half the square, do we get half the size? Well yes and no – the square is half the size, but its capacity is 1/4 of the original square.

This is a very simplistic view - but it illustrates if the environment is ‘halved’ the capacity will not. I’m setting the scene so please bear with me …..

Since most of our readers are software/web developers with global reach, we want to share with you info about an upcoming international conference where you can learn cool new coding techniques and hang out with some rock stars in the developer industry. LoadStorm is a proud sponsor of this software development conference and is offering 5,000 free virtual users in our stress testing tool to attendees!

So far in our series of Web Performance articles, we've addressed the three majors types of caching that Web server application developers can employ: server file caching, application caching, and data caching. We've looked at additional performance enhancements that Web server administrators can can activate, such as HTTP compression, file consolidation, and connection pipelining.

In this latest installment of our series, we're going a little deeper and focusing on Apache. The world's most popular Web server, Apache currently powers over 63% of sites on the World Wide Web. While Apache runs decently out of the box, development teams and system administrators should combine frequent load testing with the following tuning recommendations to ensure their applications perform well under heavy traffic.

Memory Models: Prefork vs. Worker

A Web server must be able to respond to anywhere from dozens to thousands of concurrent user requests. Each request, obviously, must be fulfilled by either a separate application process, or a separate thread running within a single process.

Apache can handle concurrent requests in two different ways, depending on which Multi-Processing Module (MPM) was selected at compile time. The default MPM on Unix is prefork, which handles each separate user request in a separate instance of Apache. The other MPM module available for Unix systems is worker, which launches fewer instances of Apache, and handles multiple requests in separate threads within each process.

Worker is the preferred MPM module for Apache under Unix, as it is faster and uses less memory. So why is prefork the default? Because any team that deploys worker must ensure that any modules that run under Apache are thread-safe. The thread-safe requirement extends to any libraries used by these modules, which makes running a programming environment such as PHP under worker tricky: while PHP is itself thread-safe, there is no guarantee that its various extensions are. Some developers, such as Brian Moon, have reported success in running PHP under worker using a minimal set of extension libraries. Your mileage, however, may vary.

Forgive me for stating the obvious, but web applications are a critical part of global business in 2011. I see no alternative other than more dependence by companies everywhere on web software and Internet infrastructure. In my opinion, all business trend data predicts greater overall web usage, more complex application architectures, and tremendous spikes in extreme traffic volumes.

Critical Applications, Yet They Aren't Getting the Investment Needed

ComputerWorld last week made a definitive statement regarding the critical nature of web applications:

Those who are unprepared are vulnerable to service outages, customer dissatisfaction and trading losses - and often when it hurts the most. Successful businesses understand the need to assure service and application availability if they want to retain customers, deliver excellent service and take maximum advantage of the opportunity their market offers.
This is not a theoretical problem - just look at the recent challenges for the London 2012 Olympics andTicketmaster. Just when everyone wants to do business with you, you’re not available.

The London Olympics site was overwhelmed by high demand for tickets and many buyers received the message, “We are experiencing high demand. You will be automatically directed to the page requested as soon as it becomes available. Thank you for your patience.”

That's a failure even if the representatives of the site said it had not crashed. Performance failure...pure and simple for the whole world to see.

Examples of performance failure like this seem to occur weekly, if not daily, somewhere in the global business universe of websites.

Transformative Moment? When Global Retailers Fail!

Recently Target.com crashed under extreme user volume. They cut a deal with a designer line of knitware (Missoni) and promoted a special sale on the morning before products were sold in stores. By 8:00 a.m. EDT, the site was crashing. The Boston Globe went so far as to say:

”...the Missoni mess could be a transformative moment in the relatively brief history of e-commerce. Retail analysts say it shows that even though online shopping has made major strides since Victoria Secret’s website famously faltered during a 1999 webcast, companies still may not always have the technological muscle to meet consumer demand for such frenzied promotions.”

When will retailers learn? When will marketing departments going to consider the technical ramifications on their campaigns and launches? When will the IT department escalate performance engineering to a high priority? When will we stop reading about sites crashing under heavy volumes of traffic?

Hopefully never! Because these stories are great examples why you need LoadStorm.

Target Inc.'s website crashed yesterday due to special promotion. Apparently, the discount retailer has cut an exclusive arrangement with an Italian luxury designer called Missoni, and it would seem that the online sale of Missoni knitwear generated enough buyers to bring the site down.

I sure would like to know how many concurrent users killed it. Wonder how many requests per second the Target site was handling with less than a 5 second response time?

Can there really be more than a few hundred knitwear aficionados that would hold Missoni goods in such high esteem? What are the odds that those few hundred would all be anxiously awaiting the online sale and access the site simultaneously?

Perhaps it was 5,000 or 50,000. The result is the same - lost revenue, bad press, unhappy customers, and brand devaluation.

A rumor has swept around the world this week through Twitter and Facebook that Jason Buksh, performance consultant, is going to be an ongoing guest blogger for LoadStorm. That rumor has apparently been substantiated! You heard it here first. We look forward to many useful insights from Jason.

So, please welcome our newest performance test expert blog contributor here at LoadStorm.com.

As his first contribution (other than his insightful interview), he sent me this funny cartoon about stress testing.

Jason Buksh is a Technical Project Manager and Performance Consultant in London, England. Jason has extensive experience with performance testing at many companies including HSBC and Siemens. He is skilled with tools such as Rational, Grinder, and Performance Studio. His certifications include LoadRunner.

We appreciate his time to share some good thoughts with us about a topic that gets us excited. Here is his interview with us.

What is your technical background?

I learned to program when I was 13 - it was a vic20, I then swiftly moved onto 6502 for the BBC micro. University studying computer science was an obvious and easy progression for me. First job was writing rendering engines (C++) for virtual reality simulators. I would describe myself as a techie at heart - I’m genuinely interested in how things work. I think having a strong and long background in IT enables me to grasp new concepts easily - which is great when I have to go into different companies and need to understand their systems quickly. I've a 2:1 in Computer Science, ISEB Practitioner and SCRUM Master Certified.

Do you consider yourself more of a software developer or QA professional?

We should get out of the habit of separating them so readily. I feel strongly that every software developer should QA their work. Its not good enough to code and then relinquish QA to another team. Its lazy, increases delivery time, wastes effort and increases cost. Everyone should be a QA professional within their own field. I think a large dedicated QA team is a good measure to the inefficiency of an IT project. I’m going to write a post on this very topic.

How much involvement do you have with load and performance testing?

My career is built on it. I’ve performance tested many mission critical and highly transactional systems. Companies like Expedia have extremely large volumes of traffic, and the performance of such a system is paramount. My experience at global financial institutions has taught me a great deal about trading platforms and the importance of milliseconds in response time.

What is the biggest change you have witnessed in the way people conduct load testing?

There is a quiet move away from Loadrunner and its going to become an avalanche. It's been underdeveloped and overpriced for a long time.

In the previous installments of our Web performance series, we've examined how developers can employ Web server caching and application caching to speed up their Web applications. In this installment, we'll see how caching entire data sets can increase platform efficiency.

What is Data Caching?

Database caching is a species of application caching that caches the result of one or more database queries in the application server's memory. For our purposes, we use application caching to refer to caching any component in an application server's memory, and data caching to refer to caching a collection of data for future querying and sorting.

There are two main approaches to data caching:

1. Data set caching. Data returned from a database is stored in a data set, an in-memory representation of data that mimics the column/row structure of a relational database.
2. In-memory databases. An in-memory database is a relational database that operates completely in a server's RAM, as opposed to storing data to a hard drive. In-memory databases can be used for fast access to data previously retrieved from a traditional, disk-based RDBMS.

Let's dig deeper into each of these.

In our first installment of this series, we examined how Web server caching is implemented. Caching stores critical pieces of information in memory or on a local hard drive for subsequent rapid retrieval. In this installment, we'll look at how Web application environments can use application caching to reduce load and increase performance.

What is Application Caching?

Application caching stores calculated components in memory for future use, either by the same user or multiple users. Because of the complex architecture of modern Web applications, application caching can produce huge gains in Web site response times.

A Web application is usually comprised of multiple architectural layers. At the simplest level, Web applications consist of the client, which runs the application (usually a series of HTML pages and associated technology); and the server, which processes data and generates output. When we zoom in on the server side, however, we find that most non-trivial applications consist of several additional layers. Typically, these layers include the UI layer, which generates the user interface (usually HTML); the business logic layer, which implements business rules; and the data layer, which stores and retrieves data from one or multiple data sources.

Back in 1986, I had no idea who H. Ross Perot was. Nor did I really care. However, he not only played a unique role in American politics back in the 1990s, but he also was instrumental in the forming of the global technology powerhouse known as CustomerCentrix, LLC. By extension, he was critical to the development of LoadStorm because CustomerCentrix is our parent company.

Therefore it is only fitting that today we honor the 80th birthday of one H. Ross Perot, founder of information technology giant
Electronic Data Systems (EDS).

EDS was purchased by General Motors in the eighties because the IT outsourcing industry was booming and because GM had a poor track record with information technology. As a student in Nashville, TN during those days it was huge news when GM announced the creation of Saturn - a new car company to remake all the old manufacturing paradigms encumbering the US Big Three. The first plant was planned for a small town just south of Nashville. It was going to be an enormously beneficial investment for the area and for the whole country. We all believed the hype! So I determined that my career as a computer scientist should start with the biggest revolution in American business, and I set about getting a job in the computer side of Saturn.

The Miami Herald published an ad for clothing congratulating the team on their 2011 NBA championship. Perhaps no one noticed. #FAIL

Whoops, I guess the editor wasn't watching the game when the Dallas Mavericks closed out the NBA Finals and took home the trophy. Congrats to the Mavs. Dirk earned the right to be in the greatest player discussions - especially when it applies to comebacks in the 4th quarter.

The huge mistake by the newspaper makes me think about how obvious web performance failure is a tremendous error. Perhaps a site owner or web application product manager can ignore the performance aspect, but their users will not. The speed and scalability of an online system has been proven statistically to directly affect its effectiveness.

Rodney Bone is a performance consultant that works for Revolution IT in Brisbane, Australia. He has graciously invested his time to share insights about performance testing with us. Please follow him on Twitter (@rod_bone) and tweet your thanks for his interview.

That's Rod in the picture at the right when he did some reserve time during the Brisbane floods over Christmas. The slow sign is just ironic.

What is your technical background?

Started as a software engineer with Accenture, where I was involved in the entire end to end SDLC including experience in the BA space.

Do you consider yourself more of a software developer or QA professional?

Now, definitely QA, I think todays generic developer is only one part of a large picture and Accenture’s model of exposing developers to the whole SDLA is one that should be encouraged by all companies.

How do you determine the load to apply to the target app during load testing?  

Ask the business, and ask as many people as applicable. Users, and BA’s have different opinions, so talk to them all. Once you nail the processes and the amount of transactions per hour run with that. Back that up with Log info as historical information is the best source of information. With new applications this is not always available.

Do you prefer using requests per second (RPS), concurrent users, or some other metric to define load?

They answer different questions. You can sometimes hit a server with the right RPS with one virtual user but that doesn’t tell you that the web server is up to serving the required amount of concurrent connections. I have tested an app where they plugged the Apache Load Balancer in out of the box and it only served 20 concurrent connections. FAIL! And you wouldn’t pick this up without concurrent users.