Checking Is Not Evil

A few days ago, over at, Cem Kaner expressed the view that manual exploratory testing had become the new Best Practice. Cem’s comments resonate strongly for me as I have recently recognized and sought to correct that in my own thinking.

Whilst on BBST Test Design last year, Cem called me out on one question where I’d answered from a position of context driven dogma rather than any kind of well thought out position. This was an invaluable lesson that made me begin to question my prejudices when selecting a test approach.

I recognized that I had begun to show a strong preference towards manual exploration over automated testing, of disconfirmatory testing over confirmatory checking. I resolved to challenge myself, and for every approach I considered, to think of at least two others so as to compare and contrast their relative merits.
Then came the moment of truth: a new project. And as luck would have it, one that would go to the very heart of this issue.
The project in question is all about data. There are many thousands of conditions that are specified detail, the data combinations expand this out to potentially billions of individual checks. I would be ignoring a significant part of my testing mission were I to disregard either checking or automation in favor of manual ET. In short, the context demands that large-scale mechanized checking form a big part of the test approach.
Does this mean that we will rely solely on checking and disregard exploration?
Does a check-heavy strategy make this any less context driven?
No, and no. As I wrote here, checking may be necessary but it is not sufficient. This project is not unique: there is plenty of opportunity for unfulfilled needs, undesired or unexpected behaviour. We will test the specifications before a single line of executable code is available. We will challenge our own understanding of the design, and seek to align the perspectives and understanding of others. We will use risk based test design and seek to reveal failures. We will accept that the specification is not perfect, and that we will need to adjust our thinking as the project evolves. We may check, we may not test manually, but the essence of what we do will remain both exploratory and context driven.
Checking is not evil, nor (as far as I am aware) did anyone say that it is. 

23 thoughts on “Checking Is Not Evil”

  1. There is no “Context-Driven dogma” that opposes the use of tools, nor is there any “dogma” that exhorts us to test “manually”, whatever it is you mean by that.

    If some people misunderstand this, then we just have to keep working to correct the misunderstanding. Testing without tools is not in any way a best practice. There are no best practices.

    But testing without thinking is not, to me, testing at all. And machines don’t think.

    Checking is not evil– checking is simply not testing. Testing involves a lot of checking, but the two are not the same thing. The point of making the distinction is to help people put automation in perspective.

    That’s the thing: perspective. Automation must be kept in its place. It’s place is not to be assigned the task of testing. Only people can do that.

    If you can point to any writing anywhere that you want to identify as “CDT dogma” which is in any way harmful or unhelpful, I’ll look at that and try to fix it or explain it to you.

    — James

    1. James, thank you.

      In this post I related an experience in which I realized that I had relied on a preference rather than thinking through a problem. Cem challenged me to think of contexts where a practice that ran counter to my preferences would be appropriate. This led me to examine my preferences at length, and I was shocked by what I saw.

      Did you ever portray ET as a best practice? No. Did Michael ever condemn checking? No. Yet somewhere along the line ET had become my own preference, an answer that I’d reach for without consideration of context: it’d become a subjective best practice. Shame on me.

      Am I alone in this? No. Whilst I can’t speak for the community at large, I have observed the same bias amongst several testers that I know personally. Cem’s recent comments resonated for me because they suggest that he’s seen something similar.

      Any community will develop its own social norms, and those norms can harden to the point of closing minds. When that happens, you have dogma. I don’t want that to happen in context driven testing, indeed the very thought is antithetical to the principles that you laid down.

      So yes, here’s to “working to correct the misunderstanding”.


      1. Okay, so it was YOUR context-driven dogma that you transcended. Good. This is an event on your personal journey.

        Some people are attracted to CDT because they like the humanism of it, without really understanding what we mean when we say there are NO best practices and that the heart of CDT is that we must gain the skills to analyze context and craft solutions that fit. In time, that will come.

        1. Mine, yes. MINE ALONE? No.

          Of course, people will be attracted to CDT for a number of reasons. Many will fall into the mental trap of subconsciously substituting one set of preferences for another.

          What can we do to encourage people to think for themselves? What might YOU do to prevent them from becoming mere Bacholytes?

          1. What would a “Bacholyte” be? Someone who attempts to predict what I will think about any situation? A good way to do that, actually, is not to predict anything, but rather to think and feel based on a simple set of premises that came to me mostly via the work of my father, Jerry Weinberg, and Cem Kaner. Those three men have had the most formative influence on me.

            Note that I am greatly influenced by Kaner, even though I have strong disagreements with him. I believe he has also been influenced by me. A lot of our collaboration was productive specifically because we have such different personalities.

            My job as a teacher is to facilitate people coming into complete ownership of their thoughts and actions. Part of how I guard against students simply copying things I say, instead of thinking critically, is that I lead them through contradictory positions and I challenge them with very difficult problems that have no easy answers.

            People who work with me long term and stick with it graduate from student to colleague (and therefore leader). Michael Bolton is an example of such a person. My own brother is an example.

            Cem once told me that one of my problems is my unconventional education, since that gives me unrealistic expectations about how other people should think. I replied by publishing my book which at one point was going to be titled “It’s Never Tool Late to Get an Unconventional Education.”

  2. I’ll agree with James Bach’s points regarding automation needing to be kept in its proper place in the scheme of things in testing. I don’t totally agree with the delineation of ‘testing’ vs. ‘checking’, but that is another discussion/debate.

    What I will support and re-enforce regarding automation (any type, and at any level) is that it is only as ‘smart’ as the person who created it. Meaning you have to use the computer between your ears first in order to use the one in front of you. It is a tool only, and how you use that tool is very important. Being able to swing a hammer does not make you a carpenter. It is how you decide to use it, how it really is used, how much it is used and in what circumstances you use it (or not).

    As an automation guy I think I have a pretty good grasp on this, and I use it to help my clients wisely use automation and the associated tools for it. That is my job as a consultant and craftsman.

    1. Jim,

      I don’t disagree with these points, and have from time to time argued passionately that automation needs to remain subservient to the testing mission rather than becoming the mission in itself.


    2. Automation is absolutely NOT as smart as the person who created it! Please do not equate people and computers. Checking can be done with a computer, but no computer technology can do testing.

      Even a dimwitted tester outperforms a computer in terms of the capacity to wonder, and to learn, and to feel, and to be bored.

      1. James,
        You are misunderstanding and misinterpreting what I am saying. I am NOT exhalting the computer/automation above a human tester. I am saying that it can only be as good as the person behind the keyboard programming it. Meaning that it cannot replace the person, but can only do what they tell it to do and only in the fashion they tell it to do its work.

        I know no one can create a HAL9000 to do automation work. Some people think they can, but that is a fantasy.

        So just calm down. You don’t have to brow beat me regarding this area.

        1. Jim, you are continuing to make exactly the same mistake.

          Automation is NOT “as smart” as the people who create it. That IS what you said. Please STOP saying it. Because people might think that you mean what you say.

          Instead of getting defensive, try using that same energy to speak with more precision.

          But if maintaining the respect of intelligent people is not something that matters to you, by all means dismiss my invitation for you to correct your misstatement.

          — James

  3. Hi Iain,

    My initial take on your post was the use of a/o checks need not be demonized by CDT proponents’. – I am also of the opinion that checking is not testing, taken in to specific context ( i.e. a data driven project ) where actual data validation is a stage of the over-all initiative then I can see where the use of an automated checking mechanism can be utilized to enhance the ability of the Testers to ‘Explore Uncertainty’

    regards Michael
    regards Michael

  4. I believe part of the problem in the discussion of “Testing vs. Checking”, and the “demonize” of checking, comes with the term it is compared to.
    We use Testing as a general description of our profession – stating that Testing is better then Checking, gives the impression that Testing as a profession has no use for Checking, and that results in a lot of confusion, just since we use that same term for two different meanings.
    If we could have found a separate term for the “Testing” part within Testing profession, it would have been easier for people to grasp the difference between these two sub-activities of testing as a profession.

    @halperinko – Kobi Halperin

    1. Kobi, this is an interesting point: testing used to describe both the superset and one subset.

      I don’t see testing as better than checking: just serving a different purpose. Is a chisel better than a hammer? It depends on what you are trying to do. Neither is better than the other, they are both simply tools. Many people see testing (the superset) as simple checking – that any fool with a hammer will do. Yet favouring a chisel (because it requires greater skill) does us no favours when a hammer is called for.

      1. The comparison is not between hammer and chisel. The comparison is between hammer and carpentry.

        Testing and checking are not two practices vying for your attention. They exist on different levels.

        I think I can do a lot more interesting and powerful things with carpentry than with a hammer. But if all you want is a paperweight, then I suppose a hammer will do.

        1. Doesn’t that just serve to illustrate Kobi’s point: that there is confusion in the use of “testing” to describe both the craft, and those skills and techniques that we use that are not checking?

          I’m more than happy to think of testing (the superset) as carpentry. As a carpenter I select the appropriate tools for the job, be that a hammer or otherwise.

          1. Iain said: “there is confusion in the use of “testing” to describe both the craft, and those skills and techniques that we use that are not checking?”

            I did not say that testing is the part of testing which is not checking. But I see where I may have created confusion. I actually said this:

            “Checking is not evil– checking is simply not testing. Testing involves a lot of checking, but the two are not the same thing. The point of making the distinction is to help people put automation in perspective.”

            Perhaps I should have said “Checking is not evil– checking is simply not the equivalent of testing; it is a subset of it…”

            It was confusing for me to say checking is not testing, because while that is true, it implies that checking is separate from testing. It’s not separate. Imagine if I said “I am not my hands!” By that I bet you would understand that I did not mean to imply that my hands belonged to something other than my body. Instead I bet you would understand it to mean that my hands are only part of, and not the same thing as, what I am as a person.

            That’s the deal with checking. Checking belongs to testing. Checking is part of testing. Checking is necessary for testing. I cannot imagine testing without checking. So, of course it’s not evil. I need it! But checking is not all there is to testing, and by splitting checking out from the rest of the testing equation, we hope to say “yes, there are some things you might automate that would help testing, but that is not the same as automating testing.”

    2. Kobi, I think you may not understand what testing and checking are. Otherwise, I doubt you would be saying what you seem to be saying.

      Testing and checking are not co-equal concepts. Just as “cutting vegetables” and “cooking” are not co-equal. If you are a professional cook, you might very well desire to distinguish between slicing things and being a skilled cook. To do this, you would not create a new word that meant cooking but was different than the label “cooking.” You would just say cooking and you would mean cooking. But you might speak of “prep work” as different than cooking.

      Similarly, there’s a difference between communicating and talking. Talking does not necessarily imply that good communication is happening.

      There’s also a difference between driving a car and pointing it. Hey, I’m sure you can come up with more examples yourself.

      Michael and I are not demonizing checking. We are using language as a tool to help people understand the clear difference between the rich social and intellectual activity of testing and the comparatively simple and highly automatable activity of fact checking (or “checking” for short). This is very important in our industry. Because when you get these activities confused with each other, you start to wonder why we need skilled testers and why someone doesn’t just make a tool to do all that testing stuff.

      Yes, we are also using the distinction to make life uncomfortable for those who seem to think that checking is a reasonable substitute for testing.

      We don’t need a different word for testing than testing. Testing is all encompassing and it includes checking within it. It is not an alternative to checking, nor is it hostile in any way to checking. However, when someone becomes obsessed with the checking part of testing such that he ignores all the rest of testing, this is when it is only good and right for us to chide him. Gently, perhaps, but chide him we must.

      1. James,
        We fully agree on the concept – there is no argue here,
        All I am trying to say is that people might not understand what you say as you meant it to be heard, and miss your point.
        I would rather have another term defining the part of testing which is not checking, so we could say something like:
        Useful Testing is finding the right balance between Checking and X.
        This more clearly says that checking is a vital part of our occupation, but we can not rely on it alone, as we will miss a large portion which is X.
        I am only saying this as I see more and more people dismiss the checking part all together, and I am sure that was not your intention.
        While you are using the finer subtleties of the English language to describe testing, you should also consider that a large portion of your readers are not native English speakers, and some may not be so observant and might not get the point as you want them to.
        (It does not matter how we say, it matters how others understood what we have said)
        The mere existence of a post which subject is “Checking Is Not Evil” clearly shows the confusion is real.
        The clearer we will put it, the less confusion we will inflect.

        When you start by saying “Checking is not evil– checking is simply not the equivalent of testing”, and especially as some say “Checking is not Testing” many will not wait to hear the ending of “But it is a subset of it…” or give this part of the sentence the right balance.

        Anyhow, I am sure you are the right person to come up with a definition which will be less confusing to more people.

        @halperinko – Kobi Halperin

        1. Kobi,

          You said: “the right balance between Checking and X”.

          That is a tall order: whilst “checking” is fairly homogenous (a variety of techniques that are used for selecting checks), X (that which is testing but is not checking) is not.

          X includes learning and exploration, it includes disconfirmatory testing, it includes modeling, it includes communication and relationship building, it includes thinking, and it is not limited to these things. X is a diverse set composed of anything that can assist your testing. As such it is hard to label; perhaps that is why it hasn’t been named.


          Addendum: If checking is machine executable, then “techniques that are used for selecting checks” falls under X as well, leaving only the act of checking itself within “Checking”.

        2. I’ve heard several people (including Prof. Jeffrey Kasser in Teaching Company lectures on Philosophy of Science) talk about philosophy as clearing out linguistic underbrush. Here’s an attempt at that.

          There’s a trap that we fall into when we ask for a term to describe “the part of testing that isn’t checking”. That’s like trying to find a label for “the part of a tree that isn’t leaves”. In fact, there are many parts of trees that aren’t leaves: twigs, flowers, branches, cores, seeds, limbs, trunks, bark, cholorplasts, cones, capillaries, DNA, pistils, stamens, fruit, stems, peels, lysosomes…

          I cook. As I cook, I do a lot of slicing. As James points out, slicing is part of cooking, but cooking isn’t slicing. It would be weird to have a single term that wrapped up all of the non-slicing parts of cooking, wouldn’t it? When we say “cooking isn’t slicing”, we don’t mean that cooking and slicing are mutually exclusive; we’re trying to point out that it would be a mistake to think of slicing as all there is to cooking, or as the most important aspect of cooking. Slicing is a very important skill, but our thinking about it should not dominate our thinking about cooking.

          —Michael B.

Leave a Reply

Your email address will not be published. Required fields are marked *