“When we uncover something…we are like investigative reporters, revealing something that would otherwise have remained unknown”. – Understanding by Design, Grant Wiggins and Jay McTighe.
At its heart, testing is about discovery and understanding: we seek to understand our stakeholders and their expectations, we manipulate and observe the behavior (and misbehavior) of the software so that we might discover how it behaves, we adjust and refine our understanding of both as we learn.
I’ve recently been reading Understanding by Design, a book dedicated to putting the understanding back into education. At times, the parallels with testing are striking: of particular note is the distinction that Wiggins and McTighe make between coverage and uncoverage.
The authors describe coverage in education as a negative term “because when content is covered the student is led through unending facts, ideas, and readings with little or no sense of the overarching ideas, issues, and learning goals that might inform study”. They liken it to a whirlwind tour of Europe, with each day being another country with little opportunity to take in much of anything. In contrast, they describe uncoverage as “finding something important”.
Let’s take a moment to look at coverage in the context of testing:
- Each form of coverage is underpinned by certain techniques, each of which has blind spots. By pursuing coverage with one we forgo opportunities to apply others; we forgo the opportunity to learn what other techniques might teach us.
- Coverage has a focusing effect. Each technique leads us to observe certain aspects of software behavior at the cost of inadvertently ignoring others, again missing opportunities to discover.
- Coverage can become a proxy for testing “progress”, resulting in an inexorable march to schedule, further curtailing opportunities to learn. How many times have you found yourself driven to complete a given set of tests at the expense of what you might uncover?
As Wiggins and McTighe point out, coverage confuses means and ends, “our teaching with any resultant learning – mere planting with the yield, or marketing with sales”.
Coverage should never be our goal in testing; it should simply be a means highlighting what remains to be done. Which of the following statements are more useful? “We’ve covered the requirements” OR “There are some risks we haven’t covered yet”. The former tells us essentially nothing about software quality or the effectiveness of our tests whilst the latter gives us something actionable.
When testing, make uncoverage your ends and relegate mere coverage to means.