Tester 1: “Are you certain?”
Tester 2: “I’m almost certain”
Tester 1: “Then you’re uncertain?”
Tester 2: “No, I, uh…I’m fairly certain”
Tester 1: “So you’re not certain?”
Tester 2: “Dammit, yes I’m certain”
What is the opposite of “certain”? You might think the answer is “uncertain”, but the English language is a tricky beast.
Perhaps adding a little definition would help. There are two forms of certainty:
- Epistemic certainty which relates to your state of knowledge and whether you hold a belief for which there are no possible grounds for doubt.
- Psychological certainty which relates to your state of mind. Call this your degree of confidence.
Epistemic certainty is an absolute1 and not a matter of degree, whereas uncertainty is scalar. They cannot be opposites. Imagine a continuum ranging from absolute uncertainty to absolute (epistemic) certainty. Any point on that scale represents a degree of uncertainty. No degree of uncertainty can be sensibly said to be the opposite of epistemic certainty, any more than it is sensible to say that any rational number is the opposite of infinity. The relationship is the same; on this continuum epistemic certainty is an unobtainable construct much in the same way that one cannot count to infinity.
What does this have to do with testing? I’ve written several times on this blog about uncertainty (here, here and here). I’ve also written a little about confidence. Having just read Duncan Nisbet’s post on Michael Bolton’s Let’s Test keynote, I think it’s time to link the two.
When we first approach an item of software, we start from a position of little knowledge. This brings with it a great deal of uncertainty. We read, we ask questions, we test. We build better models in our minds and test those models, constantly refining and enriching them. Our role as testers is to do just this, to learn, and to articulate what we have learned. A natural result of this growth of knowledge is the reduction of uncertainty. This does not mean that we increase epistemic certainty, or that we get closer to it. Moving from ten to one million does not increase one’s proximity to infinity; else infinity would not by definition be infinite.
Is our role as testers to reduce uncertainty? Ultimately yes, I believe that it is, in the epistemic sense at least. What is the value of any item of information that does not reduce uncertainty? If we provide information that has no effect on uncertainty, then we have most likely not provided any new information at all2. We might add value by providing information that increases uncertainty, by identifying an unknown that was not previously known to be unknown3 or that was previously thought to be better known than it is. However, in this sense we are not changing the balance of epistemic uncertainty, but have strayed into the realm of psychological certainty.
Psychological certainty, in contrast to epistemic certainty, is scalar in nature: one can suspect, one can be fairly confident or one can be utterly convinced. In the psychological sense, certain and uncertain are indeed opposites, and an increase in one reduces the other. So when Michael says “A key part of our service is to reduce unwarranted and potentially damaging certainty about the product”, I believe he is talking about psychological certainty4, and I’d be inclined to agree. How do we do so? By doing what we do: investigating, uncovering and revealing information that runs counter to the unwarranted certainty; in other words, by reducing epistemic uncertainty.
In testing, the danger we encounter is when we blur the distinction between epistemic and psychological certainty. “99% of the tests pass”: does this provide a case to increase our confidence? No. “We’ve found and fixed 1000 bugs”? No. A warrant might justify a belief, but we should be wary of seeing ourselves providing warrants that increase psychological certainty. We should certainly not engage in managing confidence. You may be told that one of the purposes of testing is to build confidence and that your practices need to be supportive. If you agree then you are agreeing to a scam. The most we can do is create an environment in which the confidence of our customers will live or die based on relevant information being made available to the right people when they need it. Their confidence is their business.
- 1 You might ask if I’m certain about this: my answer is no. It is entirely possible that one day some bright spark will solve the problems that have been plaguing philosophers for thousands of years, therefore I have reason to doubt this belief, and therefore I am not certain – in the epistemic sense. I might concede to being certain in the psychological sense, but that’s my problem.
- 2 Think repetitive regression testing.
- 3 A Rumsfeld.
- 4 It makes as much sense to talk about reducing epistemic certainty as it does to talk about – you guessed it – reducing infinity.