Software Testing One's Code – LoadStorm

Most people believe that if it ain’t broke, don’t fix it. Software engineers believe that if it ain’t broke, it doesn’t have enough features yet.

If you are a coder, I bet you have heard the old saying, “Programmers should never test their own code.” I’ve had several Product Managers tell me that. Of course, they had pointy hair. And maybe we were in a meeting with screaming people who were angry about a bug I had created that ruined weeks’ worth of work. Yadda, yadda.

Okay, I probably deserved some negative publicity from the incident, but does that really lead to the indisputable conclusion that just because I wrote the code that I cannot adequately apply quality assurance techniques? Can a scientist that creates a series of instructions for a complex machine not also ascertain whether or not the instructions are accurate? Is it exceedingly difficult for one person to have the intelligence and experience to examine the accuracy of an experiment regarding input and output from software? (Methinks there be too many ex words here)

But you need to be aware of potential blind spots that could cause you to miss some things. If you stay too focused on a checklist of features to test, you may not get the broad coverage of testing that you should.

I came across something very interesting that I would like to share with you. It has applicability to software testing. Without giving away the punch line, I would like you to watch a very brief video. The instructions are simple: count how many times the people in white shirts pass the basketball.

I found this after listening to a podcast from noted testing expert Dr. Cem Kaner that I downloaded on iTunes. The video is by Daniel J. Simons, and it is from Visual Cognitions at the University of Illinois. Simons is the researcher.

Remember, count only the white team passes. Please watch this video.

How many times did the white team pass the ball? Did you see anything unusual?

If you were following instructions strictly, you probably didn’t see anything of note. I didn’t either…the first time.

While the balls are getting passed around, someone in a gorilla suit walks through the middle of the group. I didn’t see it on the first viewing because I was too focused on the white shirts. Everything black was completely tuned out by my brain. You too?

How is this applicable to software testing? And why would I mention this in a blog about software engineers testing their own software?

It shows a characteristic of the human brain which could trip up someone testing software. Especially if you are a left-brain software developer. When I have written code and then tested it myself, I usually use the spec or a feature list as a guideline. I go down the list and check off each function or item on the list when I find it works. If it doesn’t work, then I log a bug. My assumptions about the specification or the feature list being exhaustive and absolute were probably blinding me to other potential problems in the software.

Cem refers to the phenomenon as “Inattentional blindness”.
Wikipedia also calls this “perceptual blindness” and describes it thus:

“…the phenomenon of not being able to see things that are actually there. This can be a result of having no internal frame of reference to perceive the unseen objects, or it can be the result of the mental focus or attention which cause mental distractions. The phenomenon is due to how our minds see and process information. Closely related to the subject of change blindness, it is an observed phenomenon of the inability to perceive features in a visual scene when the observer is not attending to them. That is to say that humans have a limited capacity for attention which thus limits the amount of information processed at any particular time.”

Anyone testing from a script or inviolable list will be susceptible to perceptual blindness. If a software engineer is focused on following a set of instructions and/or has been told what to look for, he or she is mostly likely not watching everything going on. Otherwise obvious problems or oddities are missed completely.

Since most of us developers have learned to concentrate intensely on minute details, we tend to follow instructions closely. The more we focus on the testing script, the increased probability we will miss some aspect of code deficiency. Of course, it is easy for us engineers to find and fix a null pointer dereference, but what about something more subjective that will result in failure of the user getting what they need? What about problems arising from functional dependencies? What about errors that can only be produced from particular sequencing of activities?

I am a firm believer that good programmers can be good testers. However, perceptual blindness is a reality of the human mind that should not be treated lightly by a developer that is testing his or her own code.

It goes without saying that software engineers should study testing techniques to learn fundamentals and best practices. Furthermore, when you are living for days or weeks in code that you have created, your perception of the software can be significantly different than someone that is encountering it for the first time. You are in deep; you are focused; you are intense.

I recommend that you consciously back away from the project to clear your head. Try to reset your viewpoint to that of an outsider or a novice user.

Yes, a developer can test his or her own code – but don’t do so casually. Apply sound principals of testing. And watch out for your blind spots.