In my last article, I wrote about the paradigm shift in web application architecture and why performance testers have to re-think their strategy around testing Rich Internet Applications (RIA) for performance. Web application development processes and user expectations continue to grow by leaps and bounds. Sadly, the techniques and approaches employed to test those applications have not kept up with the same growth rate. But the good news is that newer tools are coming up and methodologies are being defined to close in on that gap. Hence, it is essential that performance testers make use of them at every phase of the performance testing lifecycle.
 
Early on in the performance testing lifecycle, testers gather requirements and collecting application usage statistics is typically one of the primary tasks. In this article, I will explain how “Web Analytics tools” can be a great source of information to gather historical data about the application usage and user behavior.


 

Traditional Web Server Log Approach


Traditionally, performance testers have been relying on the web server log files to collect historical application usage data. A web server log was and still is a great source of information. They contain enormous data on web usage activity and server errors. Downloading log files from the web server and running report generation tools will help testers get meaningful info out of them. However, web server logs have their limitations. For example,

• Usage data contained in the web server logs do not include most “page re-visits” due to browser caching. For e.g. If a user re-visits a page, no request is received by the web server as the page is retrieved from browser cache. 
• 
While data contained in the Web server logs can provide insights into system behavior, it does not help much in understanding “user/human behavior”. 
• 
Web server logs do not provide user’s geographical info, the browser they used and the device/platform they accessed the application from. All of which are vital metrics to understand user behavior on the application. 

While Web Server log files are still a great way to measure user statistics, new ways to measure web traffic have propped up that provide information from a user-perspective rather than a system perspective. A large number of organizations are implementing what is called “Web Analytics Tools” as part of their Web application infrastructure. For example: Industry reports suggest that Google analytics, a leading Web Analytics tool provider is used on 57% of the top 10,000 websites.

Forgive me for stating the obvious, but web applications are a critical part of global business in 2011. I see no alternative other than more dependence by companies everywhere on web software and Internet infrastructure. In my opinion, all business trend data predicts greater overall web usage, more complex application architectures, and tremendous spikes in extreme traffic volumes.

Critical Applications, Yet They Aren't Getting the Investment Needed

ComputerWorld last week made a definitive statement regarding the critical nature of web applications:

Those who are unprepared are vulnerable to service outages, customer dissatisfaction and trading losses - and often when it hurts the most. Successful businesses understand the need to assure service and application availability if they want to retain customers, deliver excellent service and take maximum advantage of the opportunity their market offers.
This is not a theoretical problem - just look at the recent challenges for the London 2012 Olympics andTicketmaster. Just when everyone wants to do business with you, you’re not available.

The London Olympics site was overwhelmed by high demand for tickets and many buyers received the message, “We are experiencing high demand. You will be automatically directed to the page requested as soon as it becomes available. Thank you for your patience.”

That's a failure even if the representatives of the site said it had not crashed. Performance failure...pure and simple for the whole world to see.

Examples of performance failure like this seem to occur weekly, if not daily, somewhere in the global business universe of websites.

Transformative Moment? When Global Retailers Fail!

Recently Target.com crashed under extreme user volume. They cut a deal with a designer line of knitware (Missoni) and promoted a special sale on the morning before products were sold in stores. By 8:00 a.m. EDT, the site was crashing. The Boston Globe went so far as to say:

”...the Missoni mess could be a transformative moment in the relatively brief history of e-commerce. Retail analysts say it shows that even though online shopping has made major strides since Victoria Secret’s website famously faltered during a 1999 webcast, companies still may not always have the technological muscle to meet consumer demand for such frenzied promotions.”

Jason Buksh is a Technical Project Manager and Performance Consultant in London, England. Jason has extensive experience with performance testing at many companies including HSBC and Siemens. He is skilled with tools such as Rational, Grinder, and Performance Studio. His certifications include LoadRunner.

We appreciate his time to share some good thoughts with us about a topic that gets us excited. Here is his interview with us.

What is your technical background?

I learned to program when I was 13 - it was a vic20, I then swiftly moved onto 6502 for the BBC micro. University studying computer science was an obvious and easy progression for me. First job was writing rendering engines (C++) for virtual reality simulators. I would describe myself as a techie at heart - I’m genuinely interested in how things work. I think having a strong and long background in IT enables me to grasp new concepts easily - which is great when I have to go into different companies and need to understand their systems quickly. I've a 2:1 in Computer Science, ISEB Practitioner and SCRUM Master Certified.

Do you consider yourself more of a software developer or QA professional?

We should get out of the habit of separating them so readily. I feel strongly that every software developer should QA their work. Its not good enough to code and then relinquish QA to another team. Its lazy, increases delivery time, wastes effort and increases cost. Everyone should be a QA professional within their own field. I think a large dedicated QA team is a good measure to the inefficiency of an IT project. I’m going to write a post on this very topic.

How much involvement do you have with load and performance testing?

My career is built on it. I’ve performance tested many mission critical and highly transactional systems. Companies like Expedia have extremely large volumes of traffic, and the performance of such a system is paramount. My experience at global financial institutions has taught me a great deal about trading platforms and the importance of milliseconds in response time.

What is the biggest change you have witnessed in the way people conduct load testing?

There is a quiet move away from Loadrunner and its going to become an avalanche. It's been underdeveloped and overpriced for a long time.

The Miami Herald published an ad for clothing congratulating the team on their 2011 NBA championship. Perhaps no one noticed. #FAIL

Whoops, I guess the editor wasn't watching the game when the Dallas Mavericks closed out the NBA Finals and took home the trophy. Congrats to the Mavs. Dirk earned the right to be in the greatest player discussions - especially when it applies to comebacks in the 4th quarter.

The huge mistake by the newspaper makes me think about how obvious web performance failure is a tremendous error. Perhaps a site owner or web application product manager can ignore the performance aspect, but their users will not. The speed and scalability of an online system has been proven statistically to directly affect its effectiveness.

Yesterday, we examined how the new availability of elastic cloud computing has changed load and stress testing for the better. It lowers the cost, increases scalability, and facilitates running tests more efficiently. An article on load-testing.org about load and stress testing got me thinking about how our paradigms in this industry have changed and continue to evolve.

Today, we look at the reasons for why we would want to run tests more frequently and earlier in the software product lifecycle. To borrow a Chicago axiom about voting - test early and test often. It's the new paradigm.

Basilio Briceno is the Senior Developer at Naranya - one of the leading new media companies in LatinAmerica, with special focus in the mobile entertainment and mobile marketing world. He is also a Community Member of the Mozilla Foundation and Project Lead at Tlalokes PHP framework.

Basilio is or has been a college professor, a public speaker, and an independent consultant with these specialties: PHP, UNIX, GNU/Linux, FreeBSD, Solaris, Apache, IIS, Bind, Bash, Photoshop, Gimp, (X)HTML, DB/2, Websphere, JSP, JSF, Javascript, MySQL, Oracle, Perl, PostgreSQL, Postfix, and XML. Check out his personal blog site when you get a chance.

Let's start the interview.

How much involvement do you have with load and performance testing?

That's what I do everyday, and that's why companies hire me. My principal task is to be the most worried person in the company about performance testing. That's why I try to be involved in every aspect, from UI testing, to load testing, and OS tuning.

What would you say is the difference between load testing and performance testing?

To put it into a boxing metaphor, performance testing allows us to know the precision and speed of the fighter's arms and fists, the power of his punches, the resistance, velocity and movement of his legs. Load Testing allows you to know the amount of rounds the fighter is capable of competing.

What do you think is the most important aspect of load testing?

Emulating the real conditions that the application is going to be exposed to and exceeding the expectations. If the Load Testing results are superior to expectations, it is less likely that uncomfortable surprises will appear in production.

We have about 3 feet of new snow here from the storm earlier in the week. It makes for a picturesque Christmas holiday for my family.

Today I'm just going to share a summary of 2 performance testing articles that I read this morning, then I'm taking off early to enjoy time with my wife, mom, and daughters.

Performance Testing in Agile Framework

Here are a few good insights about software performance from Robert Read in his eBook entitled, How to be a Programmer: A Short, Comprehensive, and Personal Summary. Robert dedicated the book to his colleagues at Hire.com.

My favorite parts are listed here as excerpts and included below in the original context:

• Bottlenecks in performance canl be an example of counting cows by counting legs and dividing by four instead of counting heads.
• The purpose of stress testing is to figure out where the wall is, and then figure out how to move the wall further out.
• If the wall is too close to satisfy your needs, figure out which resource is the bottleneck (there is usually a dominant one.) Is it memory, processor, I/O, network bandwidth, or data contention?
• Performance is a part of usability.
• Most software can be made (with relatively little effort) 10 to 100 times faster than they are at the time they are first released.
• If a well-isolated algorithm that uses a slightly fancy algorithm can decrease hardware cost or increase performance by a factor of two across an entire system, then it would be criminal not to consider it.
• A plan for stress testing should be developed early in the project, because it often helps to clarify exactly what is expected. Is two seconds for a web page request a miserable failure or a smashing success? Is 500 concurrent users enough?
• I've made errors such as failing to provide a relational database system with a proper index on a column I look up a lot, which probably made it at least 20 times slower.
• Other examples include doing unnecessary I/O in inner loops, leaving in debugging statements that are no longer needed, and unnecessary memory allocation.
• Stress testing is fun.
• Who has particular knowledge about a component also constantly changes and can have an order of magnitude effect on performance.
• Finding the expensive I/O and the expensive 10% of the code is a good first step
• There is not much sense in optimizing a function that accounts for only 1% of the computation time.
• Each change brings a test burden with it, so it is much better to have a few big changes.

Roger Moore turns 83 today. As an actor, he is best remembered as James Bond in several movies during the 1970s where he always got his man (and his girl). James was an inspiration to all of us hot-blooded boys, and his exploits of saving the free world from evil villains was the epitome of performance.

On to today's performance testing news. Apparently IE9 is faster with HTML 5 than Firefox 3.6. Who knew?! Is this the first time in history that a Microsoft browser is the fastest at anything? Read the article below to find out.

Are you getting ready for the holiday season? Can your website handle the huge influx of traffic your marketing department is about to send your way? Tis the season to be load testing. If you aren't hammering your site and performance tuning it in the next month, it will probably be too late to prevent lost revenue.

Neil Ashizawa has written two articles in one entitled How to Test Your Web Apps to Avoid Website Downtime This Holiday Season with a second part that is called, How to remove testing headaches from holiday shopping season.

I ran into a Tweet today that said, "Is #loadtesting to know *where* it beaks, not *if*. That's it does isn't the issue, that I know where and roughly when is #load #testing". The tweet caught my eye because of the hash tag for loadtesting.

The implication, as I interpreted the 140 character bit of wisdom, is that load testing answers the questions:

1. How much load breaks a system?
2. Where does the system break under load?

Poor Performance - Your Fault?

Who is to blame for bad application performance? by Alois Reitbauer is an informative look at how developers, system architects, testers, R&D managers, and operations leaders can each play a role in poor performance of software.

While pointing the finger is a common way employees in companies invest their time, rarely does it have much ROI. My experience is that development teams, IT departments, and company executives usually don't play together very well. They don't communicate clearly or frequently to each other. It's only natural because they have their own jobs to do, and their job evaluation (i.e. bonus or raise) isn't measured by collaboration.

Perry Reed has worked in performance testing for eight years, and he has extensive experience with load testing for large software applications. Perry has applied his skills at such companies as Publix Supermarkets and Home Shopping Network.

Andrew Lombardi is a Java Software Engineer and Head Geek of Mystic Coders, LLC in Santa Ana, California. His blog provides insight into many aspects of technology.

Our thanks to Andrew for taking the time to share some information with us.

Ricardo Sueiras is a self-professed "IT professional and geek" (found that on his blog. His formal title is J2EE Systems Architect at PricewaterhouseCoopers LLP in Canterbury, United Kingdom.

Ricardo has extensive experience with web performance and load testing, and we are grateful he invested his time to share some good thoughts with us about a topic we are excited about.

What is your technical background?

I am a system architect, working in J2EE for around 10 years, with a focus on infrastructure design and architecture.

Do you consider yourself more of a software developer or QA professional?

Neither I guess - I am the guy that has to put all the pieces together, make sure it all works and performs as required. When it doesn't, I figure out why and what needs to be done!

When and why did you get into this industry?

I've been in IT for over 20 years as it has always been a passion - it is hard to think of any other industry to be in for me, and am still as interested and driven as when I started 20 years ago. I think its IT's ability to evolve, change and for new technologies to come along that help keep it fresh and interesting.

What is your specialty? Why?

I tend to specialise in open source and collaboration/social software. I have implemented over a period of years a robust process within our organisation for the adoption of open source tools, platforms and software and am heavily involved in collaboration software, initially with the first wave (cc:Mail/Lotus Notes) and now onto the new wave of social collaboration tools. I am also interested in load testing, and have worked on load testing a number of our internal web infrastructures and applications.

Bob Williams, Senior Manager e-Commerce Marketing at Harland Clarke, has a nice post about performance testing, load testing, and stress testing. He titles it Customer focused eCommerce: Volume testing techniques