Performance testing is important. Web developers never want their sites or applications to crash under load. If they do, so what? Not all of the failures below are related to high traffic, but the failures are notable because of their impact on users and the far-reaching ripple effect.
From promising thousands of customers £7 top-of-the-line computers, to nearly causing an 800-plane pile-up, ‘inconvenience’ doesn’t even begin to describe some of the most famous application failures.
We get reports of web applications crashing all the time, including one only a couple of weeks ago involving the Gmail servers crashing after ‘improvements’ to service. Everyone has experienced (or heard about) the almost constant attacks and failures of Twitter. However, the only inconvenience is usually a few rushed hours for performance engineers and a couple of hours of downtime for users.
Let’s look at some serious consequences of web application crashes:
Microsoft’s 800-plane Pile-up
Nearly exactly five years ago, a ‘design flaw’ with the way Microsoft servers were integrated into a formerly Unix system at southern California’s airport terminals almost caused 800 planes to crash.
The Microsoft servers are timed to shut down automatically after 49.7 days unless they are manually backed up every 30 days. When a new employee failed to perform the backup manually, and then the backup systems failed because of an application error, 800 planes were left circling in the air. Nobody was seriously hurt … except Microsoft’s reputation.
Amazon’s £7 computers
A data input error was the trigger for one of the most spectacular web application crashes of all time, that of Amazon in 2003. When a staff member incorrectly priced a line of new HP computers at £7 instead of £275, the servers froze and blocked customers from signing in.
Tens of thousands of orders were taken and confirmation emails sent before the server crashed – and in this case the crash was actually a service to the company!
Florida’s voting record scandal
When the servers for Florida’s e-voting system crashed in 2004, it seemed like only one more example of why the state’s voting system was untrustworthy in general. This time it was only municipal election records that were lost, but the reputation damage was a lot worse than the data loss.
UK Inland Revenue Service
The UK Inland Revenue Service (taxation department) was in the middle of a public campaign to inspire confidence in online tax return lodgment. A design flaw that meant that under load, sessions were handled improperly – and the way that was expressed was through two different users viewing each other highly confidential tax information. Oops!
8-month Rebuild
In 2002, England and Wales’ computerized records from the 1901 census were released online. The Public Records Office, however, didn’t plan for the 1.2 million visitors that were trying to access the site simultaneously at one point. The scaling problems ran deep, forcing an 8-month long process of rebuilding, load testing and performance engineering.
300,000 Personal Home Pages Lost
In China, 300,000 personal home pages, including every piece of data, was lost in a hardware crash. The raid controller and two hard disks failed simultaneously. The company was worth US $30 million only a few months before the crash. Chinaren.com can only count itself lucky they weren’t banking records!