Load Testing Slashdot

The Slashdot effect occurs when a smaller website is flooded with visitors, usually as a result of being linked to from a high profile website. The name comes from the Linux and Open Source blog http://slashdot.org, a site which handles 80 million page views a month. Many home servers or low bandwidth sites that post content interesting to one of these larger blogs have experienced significant downtime when the traffic from Slashdot suddenly migrates to their own servers. Performance engineering for the Slashdot effect is essentially engineering for a worst case scenario – but that worst-case involves more traffic than developers could ever have imagined!

We look at the Slashdot effect in terms of:

• Its magnitude
• Planning for when, not if, a Slashdot event happens
• Whether every site needs to build architecture and applications to cope with Slashdotting
• How developers and performance engineers can cope with a Slashdot event
• Some third party and minor supplementary solutions for developers trying to plan for the Slashdot effect

The Possible Magnitude of a Slashdot Event

Stephen Adler published a paper in 1999 showing the magnitude of the Slashdot effect on three papers published on the Internet. One was linked to by Slashdot.org, one by linuxtoday.com and the third by freshmeat.com. The graphs in Adlers’s paper shows an immediate enormous spike in traffic to over 50 hits per minute in each of the events, where hits per minute runs close to zero for the remaining period of the study (four months worth of data). In other studies, the Slashdot effect has been seen to increase requests to between several hundred and several thousand per minute.

The effect can most clearly be seen in the data following the Slashdot linking because Slashdot does not reproduce the full text of an article with citations, they only include a link to it. In the study, the article published in Linux Today was reproduced in full, so much of the traffic was taken up by the host servers. Copyright issue complexities are what discourages many sites from reprinting articles in full. However, for sites that merely include a hyperlink, the impact of the re-directed traffic is much more catastrophic.

What Types of Sites Fall Victim to the Slashdot Effect?

Sites like CBS News, Wikipedia and Google are hardware rich (Google, as a case in point, runs upwards of 10,000 servers). These sites are widely linked to and have built up their traffic levels over time. They have thus had the opportunity to gradually engineer performance issues and implement strategies to cope with periods of high demand. As high revenue generators, they also have the financial backing to do this. These sites do not usually suffer from being Slashdotted.

Sites that feel the full impact of an unexpected Slashdot link are usually those that:

• Are self-hosted
• Offer movie files or high resolution images
• Have dynamic content generated by inefficient methods

Slashdot members (who pay a fee for content and have first access) have occasionally rendered a site unavailable before the story is even released to the general public.

Should Everybody Engineer to Cope With Being Slashdotted?

The Slashdot effect is particularly insidious because, by nature, it occurs in sites where developers and owners usually don’t have the resources to forward-performance engineer for the scenario. If every site created their architecture from the beginning to cope with a single Slashdotting event, hardware manufacturers would exclusively fill the Fortune 500 list, and the web would be a much less profitable place for many than it currently is.

That said, website developers and owners can put some strategies and mechanisms in place to help plan for the eventuality. For sites that sell a product, that are building a particular brand, or that closely represent a company, doing capacity planning for a Slashdotting makes good business sense. If the performance risk (the worst-case scenario from a negative performance event) is judged to be greater than the cost of additional hardware or more powerful architecture, then forward-planning for a Slashdot eventuality makes sense. If the performance risk is negligible, then after-the-fact coping mechanisms, third-party application assistance and other supplemental strategies will be the most cost-effective path.

Resources??

Introduction – What is the Slashdot effect??a. The possible magnitude of a Slashdot event ( http://www.uibk.ac.at/linuxdoc/LDP/LDP/LG/issue38/adler1.html)

What types of sites fall victim to the Slashdot effect? http://en.wikipedia.org/wiki/Slashdot_effect#Extent

Testing performance for worst-case scenarios p19 and summary p5″ http://www.perftestplus.com/resources/Performance_Engineering_Strategy_T… p19 and summary p5 coping w /..ing Coping strategies after the Slashdot event?a. De-coupling components?b. Gating access?c. Redesigning the flow of transactions – coping strategies pdf p4

Third-party and supplemental solutions to the Slashdot problem (coping w slashdotting pdf p5) and http://en.wikipedia.org/wiki/Slashdot_effect#Assistance_and_prevention