When is software testing finished?

Determining when testing is completed is not always a simple matter. In any significant application, it is impossible to prove that the software will never fail. As an example, Microsoft employs thousands of professional testers for their Testing and QA Teams. These testers receive excellent training and tools. They have extensive backgrounds in software testing and quality assurance.

These MS testers are focused on every aspect of testing, and they work very hard to get products through unit testing, integration testing, and system testing. After that, potentially millions of people are given free beta copies of the software for beta testing, and many more defects are discovered.

Even after a thorough beta cycle with millions of people, bugs are again found when the product releases to the public. Microsoft is one of the premier software companies in the world, yet they are continually developing and releasing patches. Who hasn't had an "unexpected feature" pop up while using a MS product? But at some point, the product team must ship the software.

Often the decision of when testing is finished is made based on schedule constraints and marketing decisions. It SHOULD be made when all the planned tests for each of the requirements hve been validated. The more realistic decision is something like: all medium and high priority items are validated and 95% of the low priority items are validated.

Defect Injection

Defect injection is a technique that is used by mature software development organizations. Controlled defects are put into the code (on purpose). When all of these bugs have been found, then the testing team can conclude that most of the other defects have likely also been found.

An interesting paper was written about how defect injection is not the best practice in agile development methodology. Get the white paper by Tom Gilb.

Successful Testing

Some companies consider effective testing to be getting armies of people (no experience needed) to just play wiht software looking for things that don't seem right. If you have enough people work with the software, they argue, you will find all of the major problems. This works to a point, but this method has holes in it because most people will do similar things with a piece of software. Thus, the "major" problems (the ones that typical users will run across) will be found. The significant flaw in this theory is that lesser used features will likely have problems that are not found.

The Pareto Principle applies in testing as in many other places. Eighty percent of the defects will be found in twenty percent of the code/features/functions. What this means is that we need to test all of the code, but particularly we must identify and work on that twenty percent of the code that has the most problems.

The goal of testing is to find defects!

In order to find the most defects, we must use a variety of the most effective testing techniques. Be smart about test planning and employ experienced people for testing. Testing often becomes the critical path activity on a software project because no one plans for it until the software is mostly complete. Without proper planning and preparation, ramping up testing quickly enough to fully test the software is next to impossible. Therefore, on many projects, testing thoroughness is sacrificed and the software is released with undetected defects.

Just as productivity varies widely between developers and development teams, so too does the effectiveness of testers. Throwing more inexperienced bodies on the testing team is not the answer. Good project managers will hire experienced testing staff and invest in training for them. There is no substitute for quality - especially when it comes to quality assurance.