Uncoverage

“When we uncover something…we are like investigative reporters, revealing something that would otherwise have remained unknown”. – Understanding by Design, Grant Wiggins and Jay McTighe.

At its heart, testing is about discovery and understanding: we seek to understand our stakeholders and their expectations, we manipulate and observe the behavior (and misbehavior) of the software so that we might discover how it behaves, we adjust and refine our understanding of both as we learn.

I’ve recently been reading Understanding by Design, a book dedicated to putting the understanding back into education. At times, the parallels with testing are striking: of particular note is the distinction that Wiggins and McTighe make between coverage and uncoverage.

The authors describe coverage in education as a negative term “because when content is covered the student is led through unending facts, ideas, and readings with little or no sense of the overarching ideas, issues, and learning goals that might inform study”. They liken it to a whirlwind tour of Europe, with each day being another country with little opportunity to take in much of anything. In contrast, they describe uncoverage as “finding something important”.

Let’s take a moment to look at coverage in the context of testing:

  • Each form of coverage is underpinned by certain techniques, each of which has blind spots. By pursuing coverage with one we forgo opportunities to apply others; we forgo the opportunity to learn what other techniques might teach us.
  • Coverage has a focusing effect. Each technique leads us to observe certain aspects of software behavior at the cost of inadvertently ignoring others, again missing opportunities to discover.
  • Coverage can become a proxy for testing “progress”, resulting in an inexorable march to schedule, further curtailing opportunities to learn. How many times have you found yourself driven to complete a given set of tests at the expense of what you might uncover?

As Wiggins and McTighe point out, coverage confuses means and ends, “our teaching with any resultant learning – mere planting with the yield, or marketing with sales”.

Coverage should never be our goal in testing; it should simply be a means highlighting what remains to be done. Which of the following statements are more useful? “We’ve covered the requirements” OR “There are some risks we haven’t covered yet”. The former tells us essentially nothing about software quality or the effectiveness of our tests whilst the latter gives us something actionable.

When testing, make uncoverage your ends and relegate mere coverage to means.

3 thoughts on “Uncoverage

  1. Pingback: Five Blogs – 16 April 2012 « 5blogs

  2. I agree with that “uncoverage” is more important than “coverage”, however in practice we should set the goal, otherwise the quality, especially in agile process will be out of control in large scale programings. I can image the product has high quality if no one care about the coverage!

    • What goal would you suggest, and using what type of coverage?

      If quality is value, then we improve quality by finding and removing those problems that undermine value. All coverage can does for you is suggest places you haven’t looked for such problems. Each type of coverage is an abstration, each will find certain types of problem, and miss others. Slavish adherance to coverage goals can serve to focus the tester (or developer) on one set of problems at the expense of others. Unfortunately, bugs don’t behave or advertise themselves – we don’t know in advance what techniques will find them – and therefore which types and level of coverage will be useful.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>