April 16th, 2014 by Adam Sandman
requirements management test management software development
Why was it deemed necessary to automate the test management process before there were any decent tools for recording exactly what we were testing for?
The evolution of software tools to manage the software development process has been one of reverse progression, not in sophistication, or time, but terms of the stage of the development life cycle supported. Some of the earliest tools helped in the process of configuration management; managing version of software components, defining baselines from combinations of those versions, comparing baselines, and of course, helping to build the program from the defined components.
But from that point on, the introduction of new software tools tended to take on a reverse project flow. Test management tools came next, and ultimately test automation tools, both of which are used at the end of the traditional development life cycle. These tools supported the testing process by allowing tests to be formally recorded and the results saved so that software engineers could see where they needed to focus their effort, and of course also reporting test result statistics. Then, as software tools matured, the support of the development process slowly began to move backwards in the life cycle.
Some groups dabbled with context sensitive editors to help the software engineer directly, and most projects were introduced to debuggers. Moving back further still, we saw the advent of design tools made possible by the increasing sophistication of graphical displays. Once the design process could be supported with software tools, it became the goal to move further back in the process and provide analysis support, usually being combined with existing modeling and design capabilities.
Lastly, and rather oddly, we move back far enough in the life cycle to see the start of the support and automation of requirements management. Initially, people tried to use generalized tools such as spreadsheets and word processors for this, but no satisfactory solution presented itself until requirements and traceability management products came on the scene.
There are exceptions to this reverse progression, especially with tools that support cross-lifecycle functions such as process management and project management tools, but generally, this was the order of support tool maturity.
Because, broadly speaking, tools developed in this reverse lifecycle manner, it was only natural that tool integrations followed the same path. Test tools with configuration management, design tools with code generation, then requirements tools with testing tools and ultimately requirements tools with analysis and design tools, (the maturity of which is still on going.)
In some cases, the order of tool maturation is understandable; software engineers, especially while doing everything else manually, needed to be able to organize the proliferation of code components they were creating, hence CM tools. But one sequence of events seems, in retrospect, rather puzzling. Why was it deemed necessary to automate the test management process before there were any decent tools for recording exactly what we were testing for? How meaningful can tests really be if there are not clear statement regarding what the software should actually do? This seems to be very much a case of putting the cart before the horse. As has been said many times, introduce a tool into a bad process and you simply do the wrong things faster. So yes, you could test code much faster and record the results, but the test criteria were still based on badly managed descriptions of user needs. I seem to recall, (yes, I am old enough to recall) that much testing was actually to see whether the program did what the software engineer said it should, without crashing! At one time the software engineer was the one with most tools for the job, and was often the flag bearer for the end user. Then along came formal requirements tools and suddenly less technical people were able to define requirements, prioritize them and specify the content of releases.
So, for a considerable period of time, we did well organized testing, but not against requirements that were formally defined. The chicken came very much before the egg.
Footnote: Interestingly, Agile processes have given back much of the end user representation to the software and test engineers, perhaps trying to reverse the results of evolution. A tall order indeed.