May 2nd, 2014 by Adam Sandman
Let us try a dictionary definition:
“An imperfection that impairs worth or utility.”
But that just leaves us needing to define ‘imperfection’, which is not very helpful. Fortunately, the English language has absorbed so much richness from other languages along the way that we can always choose a second definition if we don’t like the first, so we also have:
“A lack of something necessary for completeness, adequacy, or perfection.”
Much better! The key word here is ‘necessary’ which leads us directly to the need for requirements for it is the requirements that define what is necessary. But as I consider this definition as it applies to software, I can’t help feeling it is still not quite right; it’s missing the imperfection element of the first definition. Let’s try a different approach.
I asked a few well respected software and systems engineering friends for their definitions. The simplest was:
“A defect exists when the software does not work as expected.”
On the face of it this seems quite straightforward, but as before, one definition leads to another. What do we mean by “expected?” My expectations may be very different from yours, especially if we are different types of user. Casual, occasional users probably look at behavior very differently to power users. Also, what might be expected by the developer may not be what is expected by the users at all. Take the message we used earlier, “The system has encountered an unexpected error and must shut down.” On the one hand the message says the error was unexpected, and yet there was sufficient expectation on the part of the software engineer that something might go wrong that it was necessary to build in a message to cover that eventuality. ‘Expected’ is far too subjective to be useful. It has the hint of ‘necessary’ which we looked at earlier, and which in turn led us to requirements.
The requirement is the standard refuge of those looking to avoid the problem of subjective expectations, so let’s try this:
“A defect is anything that does not behave in accordance with the requirements.”
After all, this is why requirements exist: not merely to help the software engineer build software that does as ‘expected’, but to also help us know what is a defect and what is not; to help testers know what to pass and what to fail. But the problem is, we can’t put everything into the requirements. For example, I know that when I press the space bar on my keyboard, I only expect one space, but if I get two spaces every other Thursday except during leap years, I would consider it a defect. How should we specify against such defects in the requirements? Making a statement such as, “The software should work according to usual expectations” is rather meaningless. We’re right back with the question of ‘expected’, not to mention, ‘what does ‘usual’ mean’? I once worked on a project to develop a software system to replace an existing multi-user product. But when the software failed to perform well under multi-user stress testing, (which had been in the test plans from day 1) one of the software engineers made the comment, “Well, the requirements don’t actually say it should be a multi-user system!”
So, now we’re left trying to decide how comprehensive the requirements should be and how much we can leave to common sense. As the old saying goes, “common sense rarely is.”
Ultimately, this large gray area of undefined ‘common sense’ means that there is no precise and simple definition of a defect which would be useful. This is one of the reasons some have tried to avoid the use of natural language in the definition of requirements and over the years a number of precise, technical methods have been proposed. But as soon as we move away from natural language to something more mathematically unambiguous, we leave the average stakeholder to the extent that any technical requirements definition model requires translation back into natural language in order that the end user can understand it, and then we’re right back where we started.
There is also another scenario. In testing I might find that the escape key takes me back to my home screen faster than by following menu options. However, if this was never specified in the requirements, is it a defect? It is certainly unexpected, but at the same time it may be rather desirable. There is only one way to find out, and this is how all dubious or questionable defects should be managed; record the behavior and let the stakeholders (or their representatives, the product owners) decide. Incidentally, this inability to be totally sure whether behavior is a bug or merely unexpected but acceptable, is another argument for iterative or Agile development. The sooner in the development process such questions can be put to the stakeholders, the sooner the behavior can be accepted and documented, or rejected and corrected, preventing proliferation of the problem into later, larger and more complex builds.
As a footnote, there has been some discussion in recent years about the use of the word, ‘bug’ as a term to describe a defect. Arguments have been put forward about the term conveying insufficient seriousness or having connotations of spontaneous appearance in the code as though nobody wrote the bug into the system. It seems to me that these arguments move the focus away from the important issue of identifying bugs and as such are simply a distraction.
A defect by any other name would smell as bad!