In the previous post in this series, I introduced regression risk and the idea that this is not a single problem but a set of different ones:

• New Bug
• Old Bug
• Zombie Bug
• Bad Build.

For each of these risks, there are a variety of mitigation strategies that fall into three categories:

• Prevention: Stop the risk from occurring in the first place.
• Confirmation: Check that things work as they did before, i.e. that the risk hasn’t occurred (Michael Bolton would call this checking).
• Disconfirmation: Seek to disprove that things work, i.e. prove that bugs have been introduced.

Let’s take each risk in turn, and consider what kinds of mitigation a project might implement.

New Bug:

• Prevention: We might review specifications of the change in the hope of preventing new bugs from being introduced.
• Confirmation: We might run old tests in order to build confidence that the new build behaves in the same way that the last one did.
• Disconfirmation: We might actively seek for new bugs by looking for failures.

Old Bug:

• Prevention: We might refactor the code to remove the added complexity – and messiness – that inevitably accompanies change piled on top of change.
• Confirmation: We might rerun tests that we created specifically to retest old bugs, in order to demonstrate that they have not crept back in.
• Disconfirmation: We might actively look for new ways that old bugs could manifest themselves.

Zombie Bug:

• Prevention: We might use reviews and static analysis to check that there is no dead code, code for which the conditions of execution can never be met.
• Confirmation: Similar to new bugs, we might run old tests to check that the software’s behaviour remains the same.
• Disconfirmation: Similar to new bugs, we might actively seek for bugs by looking for failures.

Bad Build:

• Prevention: We might implement software configuration management in order to prevent the reintroduction of old, buggy code.
• Confirmation: We might implement smoke tests to give us some reassurance that a build is sound, or – as for old bugs – rerun old bug retests.

Here’s a brief summary:

This list is far from exhaustive, and I’m not yet making any claims as to the relative merits of these various approaches.  What should be immediately clear though, is that regression testing far from the only solution to the regression problem.

But what’s wrong with regression testing?  That’s where we’ll turn our attention next.

Other posts in this series:

• Rethinking Regression, Part 1: Hard Lessons
• Rethinking Regression, Part 2: What is Regression Anyway?
• Rethinking Regression, Part 4: What’s Wrong with Regression Testing?
• Rethinking Regression, Part 5: Your Mission, Should You Choose to Accept It