Tag Archives: Best Practice


A few months ago, I met a colleague from a distant part of the organization.

“My biggest problem”, he said, “is a lack of consistency in test analysis. No two of my testers do it the same”.

“Really?” I enquired, “How do you mean?”

“Well, they document things differently, but that isn’t the problem…given the same specification they’ll come up with different tests. They approach things differently on each project. I need them to be doing the same things every time: I need consistency, I need consistent processes and methods.”

“Well, consistency can be important” I replied. “Will consistently crap work for you?”

“No, of course not. We need to be doing good testing.”

“So a consistent approach might be less important than consistent results?”

“Er, maybe. Doesn’t one imply the other?”

And therein lies the problem. With simple systems it’s reasonable to assume that you will always get the same output for a given set of inputs. Think of domestic light: you flick the switch and the light comes on. Flick it again and the light goes off. Even adding a little complexity by means of another staircase switch doesn’t add much in the way of complication. Now we don’t know whether switch-up or switch-down equates to on or off, but flicking the switch still toggles the state of the light.

If only software development was like this! Such simplicity is beguiling, and many of our mental models are rooted in the assumption that this is how the world works. It doesn’t of course.

Unfortunately, software projects are nothing like this. Imagine a lighting circuit with many, many switches. Imagine you don’t know how many switches there are, or indeed where they are. Imagine that you have no knowledge of the starting state of the circuit, and therefore which switches need flicking to get the lights on. Imagine that some of the switches are analogue dimmers rather than their binary cousins. Imagine that there are other actors who are also flicking switches or changing the configuration of the circuit. Imagine that those involved have different ideas as to which light should come on, or how bright it should be: the streetlight folk and the reading light people just can’t agree, and nobody’s talking to team ambience. Now imagine that you’ve been asked to assess the quality of the lighting once someone has figured out how to turn the damned thing on.

Now subject this scenario to a consistent set of inputs. Try to form some repeatable algorithm to reliably “solve” this problem. You can’t, it’s impossible, it’s an intractable problem. This feels a little more like the testing I know, though still a hopeless simplification.

“That,” I explained, “is why we need to emphasize skill over Method. Software projects are complex systems, and repeating the same inputs doesn’t work very well. Mechanical, repeatable processes are no guarantee of success. We need testers who are able to navigate that maze, and figure out what needs doing given.”

I made a few notes, wrote a few tweets, and got back to business. Thoughts about consistency however, never strayed far away. Then, earlier this week, they paid me another visit.

I’ve been moonlighting as a lecturer at Dalhousie University. At the start of the course I set the students an assignment (based on one of Kaner’s from BBST Test Design) to analyze a specification and identify a list of test ideas. This week’s lecture was on test design: for the first half of the lecture we discussed some of the better known, and more mechanistic, test design techniques. Then I asked the class to break into groups, compare and contrast their test ideas, and present on any similarities and differences.

“We had a few ideas in common,” said the first student “and a load of differences”. He went on to list them.

“You came up with different ideas?” I asked, feigning horror.

“Er, yes”.

“How can that be? I’ve just spent a good part of the last 3 hours describing a set of techniques that should guarantee the same set of tests. What happened?”

“Well, um, we came up with a lot of ideas that we wouldn’t have using those techniques.”

“Good, I was hoping as much.”


“Tell me why you think there were differences. Why wouldn’t these techniques have suggested those ideas?” I asked.

“Well, I guess we used our imaginations. And we’ve all got different backgrounds, so we all thought of different things.”

“Exactly: the techniques we’ve discussed are mechanistic. They might give you consistency, but they remove imagination –and your own experiences- from the test design. Now, tell me, which would you prefer: testing based on your original list of ideas, or based on the ideas of the group as a whole? Which do you think would provide better information to the project?”

“The group’s ideas” he said without hesitation.


“Well some of us focused on specific things, the overall list is better…better-rounded. We’d miss lots of stuff with our own.”

“So this inconsistency between individuals, is it a good thing or a bad thing?”

“I think it’s good. Those techniques might be useful for some problems, but by sharing our different ideas, I think we can test better.”

“Thank you” I said, mission accomplished.

Choose, Reinvent, Create

On first being exposed to context driven testing my initial reaction was indifference: I saw it as a simple statement of the obvious and couldn’t believe that anyone would approach testing in any other way. In some ways I had been lucky in my upbringing as a tester: most of my early years were spent testing solo or leading small teams. In each case I had a mandate to test, yet the means of doing so were left to me. Whenever I faced a new project I would first seek to understand the testing problem and then figure out how to solve it. In short, I learned about testing by figuring it out for myself. I’ll return to that thought later.

A few years ago, I moved into a different role. What I encountered took me by surprise:  processes proceeding inexorably from activity to activity with scant regard for need, a test manager who was so set in his ways that he refused to even talk about change, evaluation of testers by counting how many test cases they were able to perform each day. I had become exposed to factory testing for the first time, and with it came best practices: the term thrown about as marketing-speak devoid of any real meaning, used to justify lazy decisions where no thought had been applied, even uttered as if it were an invocation of the Divine; an attempt to win arguments by appealing to something packaged as self-evident and inarguable. Confronted with this insanity, I became committed to context driven testing and hardened my views on best practice.

On the subject of best practices, there is an interesting debate going on over on Huib Schoots’ blog contrasting TMap NEXT with Context Driven Testing. I’m not going to spend time on the debate itself: go read Huib’s blog. Rather, I‘ll address the central claim of TMap: that it is “an approach that can be applied in all test situations” (my emphasis). I disagree. It is a fundamental error to believe that any single approach can encapsulate the entirety of testing. Testing is about discovery and for that one requires creativity and imagination. These things simply cannot be captured in a single methodology: there is no one process for discovery. Let’s consider some examples from an earlier age, the Age of Sail. Captain Cook was, for his second voyage, commissioned to find evidence of Terra Australis Incognita, the counterweight continent thought to occupy much of the Southern Hemisphere. Cook would have made a fantastic tester. His test strategy? Circumnavigate the Southern Ocean at high latitudes and hope to hit land. How did he decide that? Do you think that he had a process model? Or that he was following some approved methodology? No, this test was inspiration pure and simple. Imagine that you are an explorer seeking new lands. What would your search strategy be? Would you sail a methodical zigzag hoping to hit land? Such an approach might work when playing Civilization, but in the real world time and trade winds would not be in your favor. No, the great explorers played hunches; they followed conjecture, myth and fable.  As they travelled they took cues from what they found, changing course to investigate distant flocks of birds, driftwood or other signs of land. Even failure was the source of great discovery: Columbus was spectacularly wrong in his estimates as to the circumference of the Earth, yet his failure to reach the East Indies resulted in the discovery of an entire continent. There are many sources of discovery, many triggers for our imagination. And imagination is best served by freeing the mind rather than corralling it with process. It may well be that some TMap testers can test effectively in any test situation: but not because of any methodology, because they are skilled and creative human beings who have sufficient common sense to break the rules.

Now, when I say that there is no one process for discovery, I do not mean to imply that process doesn’t matter. The great explorers had plenty of processes: for rigging the sails, taking a bearing, computing latitude. Captain Cook was notorious for running a tight ship, with strict diets for his crew and a cleanliness regime for crew and ship alike. Similarly, testers can apply many processes as and when the need arises. Nor is dismissing best practice the same as dismissing the practices themselves. Yes, some such practices are absurd, but others are useful in the right context. My reaction to standards and methodologies is simple: I ask myself is there something that I can steal? Testers would be well served by imitating the magpie and developing a habit of acquiring and hording every shiny thing they can find. Nor should they limit their scavenging to testing lore. The greater the range of tools and tricks we have at our disposal the better we are able to serve the needs of the contexts that we test within. As Daniel Kahneman (Thinking, Fast and Slow) puts it: “expertise in a domain is not a single skill but rather a large collection of mini-skills”. But is having a large toolbox enough? Even with the skill to wield each tool? No. What if you encounter a context for which you have no tool?

This is my other concern with packaged methodologies: by definition they contain only a finite range of tools. If the tester is constrained to using tools from a given toolbox, what happens when a new and different context is encountered? One in which no existing method is precise fit? I have routinely encountered such situations, but because I learned about testing by figuring it out for myself, I have rarely been constrained in my choices. In some cases I reinvented existing methods: such as blending UI automation, high volume test automation and random data generation to flush out troublesome reliability problems. In other cases I invented methods from scratch: such as a means of diagramming and deriving tests from the logic of retail price changes. One of my current teams is receiving significant praise because we invented an automated means of rapidly checking large volumes of data transformations. Embarrassingly, at one point I was also convinced that I’d invented domain testing, but that’s another story.

If we are to discover useful information on our projects, why limit ourselves to the test ideas suggested by a narrow set of techniques? Many projects develop myths about how the software operates, why not test those? Why not play a hunch? Even a test that doesn’t identify a problem can suggest further ideas that lead to a major discovery. Failure is powerful source of test design! If we are to excel as testers, why limit ourselves to the reach afforded us by existing methods? Why not invent new ways of testing, methods that leverage our human gifts or enhance our senses, tools that simulate difficult conditions or simply allow us to do more?

To serve our projects effectively, we must place context first.  Only by doing so can we understand need. Only through understanding need can we determine method. If we approach the testing problem from the other direction, and tie ourselves to particular methods, then we choose to place artificial limits on the value of our testing. Testers need the freedom not only to choose methods, but to reinvent or create methods as context demands.

Even in Context

1. The value of any practice depends on its context.

2. There are good practices in context, but there are no best practices.

So say the first two principles of context driven testing. But are there best practices in context? Are there practices that represent the best or only choice in any given situation?

First we need to discuss context. What is context? That’s a pretty complex question, if only because context is not a single thing. Context is not indivisible, it consists of many factors.

When exploring context, I ask a lot of questions. The following list is far from exhaustive, and is only intended to illustrate the range and diversity of factors that make up context:

  • Who are the participants of the project?
  • Who are the sponsors of the project?
  • Who is paying for the project?
  • Who are the customers of the project?
  • Who are the customers of the testing?
  • Who will use the software being developed?
  • Who else might be affected by the software being developed?
  • Where are the various stakeholders located?
  • What different organizations or organizational units do they belong to?
  • What contractual factors are involved?
  • What political factors are involved?
  • How effectively do the various participants interact?
  • What expectations do the stakeholders have for software quality?
  • What quality characteristics matter to them?
  • What other project constraints (scope, time, cost) matter to them when weighed against quality?
  • What reporting expectations do they have?
  • What methodological, standards or regulatory expectations do they have?
  • How do expectations vary between stakeholders?
  • Which stakeholders really matter?
  • What development methodology are they using?
  • What experiences do the project team have with this methodology?
  • What is the project delivering?
  • When is it expected to deliver?
  • When do people really think it will deliver?
  • What is the release strategy? Is this a one-off or will there be multiple releases?
  • What bits of the solution be given to testing when?
  • Will those bits change often?
  • What information is available?
  • How current is it?
  • How likely is it to change?
  • How accurate is it perceived to be? Will it help or hinder you?
  • What might fail and how?
  • How likely are such failures?
  • Why would a failure matter?
  • What impact would a failure have on the stakeholders?
  • How risk averse are the stakeholders?
  • What keeps them up at night?
  • How complex is the solution?
  • What technology is being used?
  • How established is this technology?
  • How experienced are the developer in using it?
  • How testable is the solution?
  • How likely are the stakeholders to listen if you need to ask for testability enhancements?
  • Who will be doing the testing?
  • Who will be testing which bits?
  • What tools do testers have at their disposal? What do they need?
  • What skills do the testers have?
  • What weaknesses do they have?
  • What influence do the testers have? Are they credible or simply ignored?
  • Why are YOU involved?

It is highly unlikely that any practice will be a perfect fit for the context you are testing in, there are always trade offs involved. For example:

  • A practice might look like a fit in terms of the quality goals of project stakeholders, yet perform poorly in terms of time and cost constraints.
  • A practice might look like a fit in terms of time, cost and quality, but be unworkable when it comes to the capabilities of the test team.
  • A practice might fit perfectly with the capabilities of the testers, yet be wholly inappropriate for revealing useful information about the solution itself.

So the value of any practice in context cannot be measured on a single scale: it will be a blend of different strengths and weakness that play to a multitude of different contextual factors. As if this weren’t challenging enough, different stakeholders will place different values on different factors:

  • One stakeholder might value time and cost over quality.
  • One might be terrified of a lawsuit and want a LOT of testing.
  • One might have an expectation that testing standards be used, and believe that time and cost should be adjusted accordingly.

In the same way that “Quality is value to some person” (Weinberg), so the value of a practice in context is its value to some person. Quality is subjective, no less so the quality of testing practices. An individual stakeholder might perceive that a given practice is best – for them – in context, but there are no absolutes that will apply to all stakeholders.

There are no best practices, even in context.