Is unit testing doomed?

Most organizations that try unit testing usually find the return is well worth the cost, so why is the approach looking like running out of steam?

By Andrew Binstock


--------------------------------------------------------------------------------

The agile revolution that began in software development in the 1990s has been inexorably making its way into mainstream IT organizations

Today, one of the most adopted agile practices is unit testing, where developers write hundreds of small tests for exercising their own code. Although the benefits of unit testing are widely recognized, there's growing evidence that unit testing might have reached its high-water mark and be entering a period of stagnation or even decline.

Signs that unit testing has hit a wall include the high-profile collapse of Agitar, which was known widely for a good, if expensive unit-testing product. CEO Jerry Rudisin told the SD Times that Agitar's collapse (it was bought by McCabe Software in June was due simply to the fact that "unit testing hasn't taken off as a mainstream practice."

Joe Ponczak, who heads up Condign Software, the makers of automated unit test generation tools and metrics dashboards, says he believes the Agitar failure was brought on by slower-than-desired growth, "not at the level expected from a Silicon Valley startup."

Andrew Glover, president of the agile-development consulting firm Stelligent, blamed Agitar's high price. "Only very large enterprises could afford it. And it was hard to justify paying those prices for a product that simply automated what developers could already do for themselves with free tools," he says.

Why unit testing may be losing its allure

Of course, Agitar is just one company, and its experience is hard to translate as an overall trend -- especially because the major tools used in unit testing are available as free open source, such as JUnit and TestNG for Java and the xUnit frameworks for other languages, as well as the free code-coverage tools Cobertura and Emma. Most of the commercial products focus on automatic generation of unit tests and coverage analysis.

But consider the typical experience of Stelligent's Glover when he consults on development approaches with clients: "When I go into sites, I would love to turn them on to the benefits of the TestNG test framework," he says, "but most organizations do so little unit testing, if any, that I stick with the simple, well-known xUnit tools, so as not to create confusion. Most organizations are just not ready to learn anything more than basic unit testing -- if that."

While languages such as Python and Ruby have active unit-testing cultures, there are signs that, in the Java community at least, resistance has built up against the unit testing idea. I posted the question on unit testing's future in my blog recently, and got several hundred responses. Among those, I identified two major threads of concerns.
One issue is that unit testing seems to offer little value.

"The problem with unit testing is that it increases the cost of development before you know whether what you developed is what you needed. Afterwards when you have a working solution, there's little interest in writing unit tests as what you have works, and your management wants you to move on," wrPeterB.ote one anonymous poster. "Many developers and managers mistakenly think that by cutting testing, they well get feature X out faster. What they don't realise is that you are robbing Peter to pay Paul," wrote Peter B.

The other issue was annoyance over evangelism, which is at the core of much of the agile movement (which began with a signed manifesto and a formulated creed). "Please, please, please, no more evangelism!!" wrote Gabriel C. A segment of the agile development community has adopted unit testing in a radical way, insisting that tests be written before the code.

This approach, called test-driven development, is intended to clarify design before committing to code. Its proponents have been zealous in promoting test-driven development as the best way to program, but for many developers that evangelism has become so overbearing that it makes unit testing as a whole unappealing.

Why you should give unit testing a second look

Unfortunately, these reasons conspire to obscure the benefits of unit testing, so it is frequently rejected at sites in favor of traditional testing, rather than being viewed as an important adjunct to it. That's a mistake. Unit testing has four significant benefits that, in my experience, all organizations that adopt it recognise almost immediately: Defects are identified nearly immediately, and because unit testing isolates their cause and effect, they are also resolved more quickly. As a result, you can be more confident in the schedule. You also get better code-level documentation.

But the first and principal result of unit testing is better code. In the traditional manner of development, code is written and quickly checked by the developer against a narrow set of criteria. Once it works, the developer moves on to other coding. Whether that routine works later on is a question that the developer faces only when examining the larger software package or during the testing cycle. At that point, the developer might easily spend hours with a debugger figuring out why the routine is unexpectedly balking.

These long debug sessions are terribly expensive to projects on several levels. As there is no method for predicting how long it will take to locate and resolve a bug, nor what effects that bug fix will have, there is no way to accurately project schedules.

Compare this to the unit testing approach in which the writing and constant running of tests immediately identifies a routine that is not working. Because of this immediacy, the developer can be essentially certain that the most recent change is the cause of a newly discovered defect.

Thus, cause and effect are tightly mapped, and the code can be fixed immediately, without long hours in the debugger trying to find the fault. Moreover, when a bug is fixed, the developer can run the entire set of unit tests to make sure no other routines are now failing due to the previous fix.

Another result of unit testing is increased confidence in the code. Under traditional testing, with each extensive debugging session, confidence in the code tends to diminish. After several such sessions, the manager has no real way of knowing whether the major bugs have been found, nor how many more debugging sessions are in store. So the final result is delivered with the hope, rather than the conviction, that it works correctly.

Unit testing ensures that defects are resolved as coding progresses, making the project schedule easier to gauge, and allowing you to more accurately project completion dates. And because unit tests capture how the developer expects a given unit to be used -- as well as what untoward data or events it might encounter that need to be accounted for -- you get better documentation.

Because the primary tools for unit testing are free and opensource, the initial costs for trying and adopting unit testing are low. In addition, unit testing can be used with any form of development philosophy, so adoption is nondisruptive.

How to view the costs of unit testing

For all of its benefits, unit testing does have costs.

The first cost is that it takes time and skill to write good unit tests. The skill is quickly learned if the organization buys into the concept. But the time spent is a harder sell, because managers often equate time spent on testing with time stolen from writing code. They forget about all the debug time that is later spent, time that unit testing would reduce by placing that work up front, when the testing is easier and the effort to fix the code is less.

A second cost is that for unit testing to really deliver, all developers need to use it. And that requires upfront training and management participation, at least to start. Unit testing still delivers benefits to developers in isolation, of course, but the ability to see what is changed in the larger project due to a change in a specific routine is lost.

A third cost is that when a software base changes in an important way, the unit tests must be discarded or substantially rewritten. In other words, the innate cost of making large-scale changes to a code base increases if unit tests are part of the mix. But it's easy to overstate this cost. After all, large changes in the middle of the coding phase should not occur often. And when they do, if any of the old code in the changed components is reused, the associated unit tests still provide value.

Most organizations that see past these concerns and actually try unit testing usually find the return is well worth the cost, and typically expand their unit testing efforts. Still, they have to get over that initial resistance. And that's why, Glover remarks, "Unit testing is likely to remain a niche solution, found at organizations that really understand its value and progressively adopted by sites that can no longer stand the long debug cycles."