The Art of Software Testing

I have been reading a classic book in our industry: The Art of Software Testing by Glenford Myers. It has a copyright of 1979, so it is celebrating its 30 year anniversary. That’s 300 years old in Internet years. It was revised about 5 years ago to include updated material on software testing tools, web applications, and newer methodologies (e.g. XP). Unfortunately, I only have the original.

Is an old book like this still relevant in the world of cloud computing, Wii, iPhones, and Twitter? Yes, I believe it is. The mere fact that it is still in print is a testament to the thousands of software professionals that have gotten great value from Myers’ insights.

The content of the book isn’t as ground-breaking as it was back in “the day”, but there are some great principals presented that are still germane. We would do well as web application developers and software testing professionals to heed some of the sage wisdom of the old mainframe crowd. Yep, I’m one of those old farts that know what a switch register was, learned to think in octal, and understood why COBOL included a mechanism for sequencing cards. Perhaps I’m just being sentimental.

It is especially hard for software developers to read a whole book about testing. I know…I’ve been there. From the time I started coding in 1980 until probably about 1998, I never read a book on software testing. We don’t have time because we are trying to hit deadlines on code releases. We are getting hammered daily by project managers that need to show some tangible progress to their boss or stakeholders. Testing?! Testing?! We don’t need no stinking testing! (my apologies to Mel Brooks).

So I will try to help those of you who will not read the book. I’ll write a detailed review and share the nuggets of gold with you that I found in this ancient text.

Complexity Illustration

First, Myers uses a form of participatory learning where he gives a simple assignment for the creation of all possible test cases needed to adequately test a program. That program determines if 3 input values for a triangle represent a scalene, isosceles, or equilateral triangle.

To expose all possible errors, it would take 14 test cases. However, in his study experienced programmers only averaged a score of 7.8.

His conclusion is still very pertinent today: testing even a trivial program is a difficult task that requires many more permutations than is apparent (even to skilled professionals). Testing is harder than it seems.

I like how he sets this up as the introduction to his concepts before “lecturing”. By using this self-assessment, and because almost no one recognizes all the permutations, it is a splash of cold water in the face. It forces us to take note and stop downplaying the challenge of testing.

Good testing may be harder than good coding.

Testing “Completely” Defined

Second, the economics of testing demand that you must choose what you want to test. You can’t test everything. The reality of software testing is that you must allocate limited resources at your command maximize the discovery of defects.

The goal of running a test is to uncover a bug. Most programmers aim for test results that proves there are no bugs. In Myers’ mind, this sabotages the true objective of testing:

“Testing is the process of executing a program with the intent of finding errors.”

He points out that testing isn’t, “establishing confidence that a program does what it is supposed to do.”

The definition of testing is not merely semantics; rather, the psychological impact on humans of trying to demonstrate “no errors exist” versus “errors exist” has a profound statistical impact on defect discovery.

According to Myers, “…testing is a destructive process, even a sadistic process.” While most people have a constructive outlook on life, professional testers must attack and rip things apart. Success in testing can only be achieved when errors are found. As a programmer, I have had many “successful tests” where my code hit end of job (old school term) with no errors. That’s what I considered successful. But Myers states that my test run was unsuccessful from the tester’s point of view.

Economics of Testing

It is too difficult and expensive to exhaustively test even a relatively small piece of code. Automated testing tools have made our tasks much easier, especially for repetitive regression testing and load testing, but it is impractical or impossible to find all the errors in a program – no matter how trivial the program.

To support this conclusion, Myers discusses the black and white box testing strategies in context of economics. Finding all errors using a black box test demands that we use every possible test case. All permutations of input data, both valid and invalid, must be tried.

Using the triangle example above, a tester would need to create test cases for all valid triangles up to the maximum integer size, which produces an enormous number of test cases, and those are not exhaustive. He goes on to show mathematically that you need an almost infinite number of test cases to detect all errors. It’s just not feasible on trivial programs. It is beyond consideration on a typical web application because the testing would cost more than a company could ever recover in revenue.

It reminds me of Calculus class in college. Thinking of limits and how X approaches infinity…. That’s the economic analysis of finding all errors – IT IS IMPOSSIBLE. Here are 2 implications of this impossibility:

1. “One cannot test a program to guarantee that it is error-free.”
2. A fundamental consideration in program testing is one of economics.

In summary, The Art of Software Testing may not be as “fresh” as many of the development books you find in Borders, but it stands the test of 300 Internet years as an outstanding primer to software testing fundamentals. LoadStorm and a few hundred other test automation tools can be a tremendous help to you when attempting to discover defects in your web applications. Still, these tools are useless if you don’t understand the basic concepts and goals of sound software quality assurance.

I recommend this book to all developers and testers. Yeah, I know…programmers won’t have time to read it.