In this email interview, software testing professional James Christie, BA MSc MBCS CITP, and ISEB accredited Software Testing Practitioner, talks about his views on usability testing, load testing, test automation, and off-shoring.
My thanks to James for agreeing to share his expertise with us. James has 24 years of IT experience, and he is dedicated to providing professional testing services to his clients in order to deliver high-quality software. I follow James in Twitter and have enjoyed his insights into the testing industry.
Living in Scotland, James has worked with large global organizations such as IBM, and he provides consulting services as leader of Claro Testing Ltd. His consulting includes setting testing strategy and budget, writing test plans, supervising test execution. and creating testing processes.
James has written an article for the latest edition of Testing Experience on the limitations of the V Model, and is currently writing another about the implications of outsourcing usability testing. He believes, and I agree with him, that usability is a typically neglected critical attribute of successful software.
And now to the interview. Because of the length, this interview will be split into three blog posts. Here is the first:
Thank you for participating in our interview and sharing your testing expertise with us. Please tell us a little about your technical background.
James: I started in IT as a Cobol programmer, but I very quickly moved into programming in SAS and CMS Exec2. Since then I’ve worked as a computer auditor, business analyst, project manager, information security manager and test manager. Sadly I’ve not had the chance to do any serious programming for about 10 years. I was pretty hot with the IBM mainframe languages and utilities, but I’ve never had the time to get as deeply involved in PC or web technology.
It’s a shame because I used to love sitting in front of a blank screen with the cursor blinking in the command line! The SAS procedural language was also great fun to work with. It was very powerful and flexible. You could write macros that would generate code dynamically so that the program was writing its next steps as it went along; the data writing the code, rather than the other way round. I worked as a computer auditor for a while, on fraud investigations, and that gave me the chance to do some really fun programming.
When and why did you get into software testing?
James: In 1996 I joined IBM when the IT department of the large insurance company where I worked was outsourced. IBM appointed me as a test manager on the Y2K programme for that client. I was told it was because of my experience as a computer auditor and project manager. They thought it meant I had the right questioning attitude. Also, I knew the applications I was going to be working on. There was a real buzz about working on Y2K. Other people might have thought it was boring, just getting date fields changed, but we knew that our clients’ systems would fall in a heap taking the company down with them if we didn’t fix them. We also knew there wasn’t enough time to do everything, so we had to take calculated risks and hit the sensitive parts hard with our testing while some applications weren’t tested at all.
What is your speciality? Why?
James: My main testing interest is incorporating usability engineering techniques into development and testing. I’m also interested in the wider topic of testing processes. I’m intrigued by the way even large companies (who should know better) have a set of testing processes that don’t hang together, or which are plainly counter-productive. For example, the company guidelines and culture encourage people to do things that make testing less effective.
Perhaps my main speciality is the ability to keep asking, “but why are you doing it that way?” without worrying that I might be asking a stupid question. I think a reluctance to ask that question is very damaging in business. It’s a pity there weren’t more bankers asking that about the shiploads of 2nd hand sub-prime mortgages they were buying in. But I digress.
What are the most critical elements of usability testing?
James: Perhaps the most important thing is not being dependent on usability testing. You could argue that there’s no such thing. When people hear “usability testing” they think of something that’s done at the end of the development. Usability testing that is left till user acceptance testing is pretty pointless. It just draws attention to flaws that aren’t going to get fixed because the implementation express is at full steam and nobody’s going to stop it for defects that can wrongly be dismissed as cosmetic and for which there’s probably a workaround.
Usability testing has to be hooked into the development process. It has to take place early enough to allow alternative designs to be evaluated. So you could say it isn’t really testing at all, it’s just good design. Users must therefore be involved early and often. You can’t do your evaluation with the developers. They’re geeks who look at applications in a completely different way from typical users. However honest they might be with their evaluation, you really can’t trust their judgement.
How do you see usability testing evolving over the past few years? the next few years?
Usability testing used to be the preserve of the HCI (human computer interface) professionals. They regarded themselves as an offshoot of psychology, a breed apart from IT professionals. Unfortunately, they operated in a ghetto where they had relatively little influence. Most applications were developed without even a cursory glance in their direction. I wrote an MSc dissertation on this very topic, so it’s hard to stop myself going on and on ……
I think that over the last 10 years there has been an increasing acceptance that the HCI people have to move into the heart of the software engineering profession, sacrificing their independence but acquiring real influence. They have to be seen not as a separate profession but as specialist members of our own profession. They have to influence the development process and help ensure that usability is built into the whole process and not just tagged on at the end (see previous answer).
I think, and hope, that acceptance and influence will continue, and that software engineers will take the involvement of HCI professionals for granted in the same way as they consult architects, database managers and network specialists. Developers will also have to learn some of the techniques of HCI. That will never come easily for them, and I hope that test managers will be proactive in helping them and helping HCI get involved.
Also, I think that there will be more emphasis on remote usability testing, especially with so much development work being done offshore. There are more tools available that help produce better results from testing where the moderator and user are in separate locations, and also where there is no moderator directly involved. That in turn allows a larger number of users to be involved, and it means they can take the tests in their usual location rather than a lab. This is probably going to mean crowd-sourcing will be more attractive, i.e. just get something out there and encourage as many people as possible to have a go at it. I’ve got my doubts about that. You’ve got to be careful. There’s a danger that the testers will be self-selected and atypical of the users you’re actually targeting.