Tag Archives: Context


A few months ago, I met a colleague from a distant part of the organization.

“My biggest problem”, he said, “is a lack of consistency in test analysis. No two of my testers do it the same”.

“Really?” I enquired, “How do you mean?”

“Well, they document things differently, but that isn’t the problem…given the same specification they’ll come up with different tests. They approach things differently on each project. I need them to be doing the same things every time: I need consistency, I need consistent processes and methods.”

“Well, consistency can be important” I replied. “Will consistently crap work for you?”

“No, of course not. We need to be doing good testing.”

“So a consistent approach might be less important than consistent results?”

“Er, maybe. Doesn’t one imply the other?”

And therein lies the problem. With simple systems it’s reasonable to assume that you will always get the same output for a given set of inputs. Think of domestic light: you flick the switch and the light comes on. Flick it again and the light goes off. Even adding a little complexity by means of another staircase switch doesn’t add much in the way of complication. Now we don’t know whether switch-up or switch-down equates to on or off, but flicking the switch still toggles the state of the light.

If only software development was like this! Such simplicity is beguiling, and many of our mental models are rooted in the assumption that this is how the world works. It doesn’t of course.

Unfortunately, software projects are nothing like this. Imagine a lighting circuit with many, many switches. Imagine you don’t know how many switches there are, or indeed where they are. Imagine that you have no knowledge of the starting state of the circuit, and therefore which switches need flicking to get the lights on. Imagine that some of the switches are analogue dimmers rather than their binary cousins. Imagine that there are other actors who are also flicking switches or changing the configuration of the circuit. Imagine that those involved have different ideas as to which light should come on, or how bright it should be: the streetlight folk and the reading light people just can’t agree, and nobody’s talking to team ambience. Now imagine that you’ve been asked to assess the quality of the lighting once someone has figured out how to turn the damned thing on.

Now subject this scenario to a consistent set of inputs. Try to form some repeatable algorithm to reliably “solve” this problem. You can’t, it’s impossible, it’s an intractable problem. This feels a little more like the testing I know, though still a hopeless simplification.

“That,” I explained, “is why we need to emphasize skill over Method. Software projects are complex systems, and repeating the same inputs doesn’t work very well. Mechanical, repeatable processes are no guarantee of success. We need testers who are able to navigate that maze, and figure out what needs doing given.”

I made a few notes, wrote a few tweets, and got back to business. Thoughts about consistency however, never strayed far away. Then, earlier this week, they paid me another visit.

I’ve been moonlighting as a lecturer at Dalhousie University. At the start of the course I set the students an assignment (based on one of Kaner’s from BBST Test Design) to analyze a specification and identify a list of test ideas. This week’s lecture was on test design: for the first half of the lecture we discussed some of the better known, and more mechanistic, test design techniques. Then I asked the class to break into groups, compare and contrast their test ideas, and present on any similarities and differences.

“We had a few ideas in common,” said the first student “and a load of differences”. He went on to list them.

“You came up with different ideas?” I asked, feigning horror.

“Er, yes”.

“How can that be? I’ve just spent a good part of the last 3 hours describing a set of techniques that should guarantee the same set of tests. What happened?”

“Well, um, we came up with a lot of ideas that we wouldn’t have using those techniques.”

“Good, I was hoping as much.”


“Tell me why you think there were differences. Why wouldn’t these techniques have suggested those ideas?” I asked.

“Well, I guess we used our imaginations. And we’ve all got different backgrounds, so we all thought of different things.”

“Exactly: the techniques we’ve discussed are mechanistic. They might give you consistency, but they remove imagination –and your own experiences- from the test design. Now, tell me, which would you prefer: testing based on your original list of ideas, or based on the ideas of the group as a whole? Which do you think would provide better information to the project?”

“The group’s ideas” he said without hesitation.


“Well some of us focused on specific things, the overall list is better…better-rounded. We’d miss lots of stuff with our own.”

“So this inconsistency between individuals, is it a good thing or a bad thing?”

“I think it’s good. Those techniques might be useful for some problems, but by sharing our different ideas, I think we can test better.”

“Thank you” I said, mission accomplished.

Choose, Reinvent, Create

On first being exposed to context driven testing my initial reaction was indifference: I saw it as a simple statement of the obvious and couldn’t believe that anyone would approach testing in any other way. In some ways I had been lucky in my upbringing as a tester: most of my early years were spent testing solo or leading small teams. In each case I had a mandate to test, yet the means of doing so were left to me. Whenever I faced a new project I would first seek to understand the testing problem and then figure out how to solve it. In short, I learned about testing by figuring it out for myself. I’ll return to that thought later.

A few years ago, I moved into a different role. What I encountered took me by surprise:  processes proceeding inexorably from activity to activity with scant regard for need, a test manager who was so set in his ways that he refused to even talk about change, evaluation of testers by counting how many test cases they were able to perform each day. I had become exposed to factory testing for the first time, and with it came best practices: the term thrown about as marketing-speak devoid of any real meaning, used to justify lazy decisions where no thought had been applied, even uttered as if it were an invocation of the Divine; an attempt to win arguments by appealing to something packaged as self-evident and inarguable. Confronted with this insanity, I became committed to context driven testing and hardened my views on best practice.

On the subject of best practices, there is an interesting debate going on over on Huib Schoots’ blog contrasting TMap NEXT with Context Driven Testing. I’m not going to spend time on the debate itself: go read Huib’s blog. Rather, I‘ll address the central claim of TMap: that it is “an approach that can be applied in all test situations” (my emphasis). I disagree. It is a fundamental error to believe that any single approach can encapsulate the entirety of testing. Testing is about discovery and for that one requires creativity and imagination. These things simply cannot be captured in a single methodology: there is no one process for discovery. Let’s consider some examples from an earlier age, the Age of Sail. Captain Cook was, for his second voyage, commissioned to find evidence of Terra Australis Incognita, the counterweight continent thought to occupy much of the Southern Hemisphere. Cook would have made a fantastic tester. His test strategy? Circumnavigate the Southern Ocean at high latitudes and hope to hit land. How did he decide that? Do you think that he had a process model? Or that he was following some approved methodology? No, this test was inspiration pure and simple. Imagine that you are an explorer seeking new lands. What would your search strategy be? Would you sail a methodical zigzag hoping to hit land? Such an approach might work when playing Civilization, but in the real world time and trade winds would not be in your favor. No, the great explorers played hunches; they followed conjecture, myth and fable.  As they travelled they took cues from what they found, changing course to investigate distant flocks of birds, driftwood or other signs of land. Even failure was the source of great discovery: Columbus was spectacularly wrong in his estimates as to the circumference of the Earth, yet his failure to reach the East Indies resulted in the discovery of an entire continent. There are many sources of discovery, many triggers for our imagination. And imagination is best served by freeing the mind rather than corralling it with process. It may well be that some TMap testers can test effectively in any test situation: but not because of any methodology, because they are skilled and creative human beings who have sufficient common sense to break the rules.

Now, when I say that there is no one process for discovery, I do not mean to imply that process doesn’t matter. The great explorers had plenty of processes: for rigging the sails, taking a bearing, computing latitude. Captain Cook was notorious for running a tight ship, with strict diets for his crew and a cleanliness regime for crew and ship alike. Similarly, testers can apply many processes as and when the need arises. Nor is dismissing best practice the same as dismissing the practices themselves. Yes, some such practices are absurd, but others are useful in the right context. My reaction to standards and methodologies is simple: I ask myself is there something that I can steal? Testers would be well served by imitating the magpie and developing a habit of acquiring and hording every shiny thing they can find. Nor should they limit their scavenging to testing lore. The greater the range of tools and tricks we have at our disposal the better we are able to serve the needs of the contexts that we test within. As Daniel Kahneman (Thinking, Fast and Slow) puts it: “expertise in a domain is not a single skill but rather a large collection of mini-skills”. But is having a large toolbox enough? Even with the skill to wield each tool? No. What if you encounter a context for which you have no tool?

This is my other concern with packaged methodologies: by definition they contain only a finite range of tools. If the tester is constrained to using tools from a given toolbox, what happens when a new and different context is encountered? One in which no existing method is precise fit? I have routinely encountered such situations, but because I learned about testing by figuring it out for myself, I have rarely been constrained in my choices. In some cases I reinvented existing methods: such as blending UI automation, high volume test automation and random data generation to flush out troublesome reliability problems. In other cases I invented methods from scratch: such as a means of diagramming and deriving tests from the logic of retail price changes. One of my current teams is receiving significant praise because we invented an automated means of rapidly checking large volumes of data transformations. Embarrassingly, at one point I was also convinced that I’d invented domain testing, but that’s another story.

If we are to discover useful information on our projects, why limit ourselves to the test ideas suggested by a narrow set of techniques? Many projects develop myths about how the software operates, why not test those? Why not play a hunch? Even a test that doesn’t identify a problem can suggest further ideas that lead to a major discovery. Failure is powerful source of test design! If we are to excel as testers, why limit ourselves to the reach afforded us by existing methods? Why not invent new ways of testing, methods that leverage our human gifts or enhance our senses, tools that simulate difficult conditions or simply allow us to do more?

To serve our projects effectively, we must place context first.  Only by doing so can we understand need. Only through understanding need can we determine method. If we approach the testing problem from the other direction, and tie ourselves to particular methods, then we choose to place artificial limits on the value of our testing. Testers need the freedom not only to choose methods, but to reinvent or create methods as context demands.

Exploring Context

Previously on Exploring Uncertainty…


Test strategy isn’t easy. If you are facing the challenge of starting a new project and developing strategy, then I have bad news for you:

Every project is unique, there is no straightforward process for developing strategy, and much of what you will read on the subject is about test strategy documents rather than test strategy.

The good news is that developing test strategy can be fascinating and rewarding.

Test strategy can be described as consisting of two parts: situational analysis and strategic planning. These should not be confused with steps in a sequential process: you will often find yourself progressing each in parallel or jumping back and forth between the two. In this post I will focus on situational analysis.

What is Situational Analysis?

Situational analysis is, well, about analyzing your situation! It’s about understanding your context and in particular identifying those factors that need to motivate your test strategy. There are a number of resources that can help guide you as to the kinds of information you should be looking for when doing so: start with the Heuristic Test Strategy Model.

This model is useful, but without skill it is nothing. To use it well, you will need to be adept at obtaining, processing and using information. Here are some skills, techniques and a few tips that I’ve found helpful.  Some of this might be useful, some of it not: you, your context and your mileage will vary.

If you’d like to add to the list, I’d love to hear from you.

Ask lots of questions

You are a tester: information is currency. To get information you need to ask questions, and be effective at doing so:

  • Ask open questions to elicit information.
  • Learn to love Kipling’s “honest serving men”.
  • Ask closed questions to test your understanding.
  • Encourage elaboration with non-verbal cues.
  • Use silence, so that others have to fill it.
  • Adapt, and follow promising leads.
  • Know when to quit when a trail goes cold.
  • Use “Five Whys” to get to causes.
  • Use models, checklists and questionnaires to remind you of things to ask. Just don’t let them limit your thinking.
  • Listen to what remains unsaid; be alert for the elephant in the room. 

Build relationships

Testing is dependent on good relationships: the better your relationships, the better the intelligence you will gather, and the easier it will be when you have to share bad news (you’re a tester after all). You need to build relationships:

  • Connect regularly, buy lunch or coffee, call “just for a chat”.
  • Develop a genuine interest in people.
  • Treat every interaction as an opportunity to learn.
  • Treat every interaction as an opportunity to build shared understanding.
  • Explore common interests and genuine differences.
  • Be prepared to keep secrets, or at least declare when you can’t.
  • Do random favours, asking nothing in return.
  • Be prepared to ask for help, and repay your debts.
  • Be honest, bluffing is for fools: never be afraid to say “I don’t know, but I can find out”.
  • Learn to relate to other disciplines, how to talk to PM, Developers…

Use diverse sources of information

Not everything useful comes in the spec, and not everything in the spec will be useful. Search far and wide for relevant information:

  • Use project documentation as a starting point not an ending point.
  • Are there user guides, manuals, online tutorials?
  • Talk to people who have used, supported or otherwise worked with the software.
  • Visit the workplace and observe how it will be used.
  • Google it.
  • Is there a website?
  • Is there a Wiki?
  • Are people talking about it on forums, Facebook or Twitter?
  • Ping your network; has anyone used anything similar?
  • Has the software, or its competitors, ever been in the traditional media?

Absorb and synthesize large sets of information

You will probably be overwhelmed. To deal with learning curves that look more like walls, you will need to be able to deal with information overload:

  • Learn to speed read.
  • Set a goal for each reading session – look for specific things.
  • Allocate some time for random facts.
  • Look for things that surprise you, that run counter to the opinions you are already forming.
  • Prioritize, but keep a list of things to come back to.
  • Look for relationships, patterns and connections.
  • Use sticky notes to cluster and rearrange facts.
  • Keep a notebook handy: you may have insights in the most unusual places.
  • Don’t just read it, model it: use mind maps, flow charts, context diagrams, decision tables etc…
  • Get others involved in the process: they will see connections you can’t.

Learn and keep learning

Every project is different, and each one will present new challenges. Rise to this challenges by building an arsenal of skills:

  • Be a magpie for every skill and technique you can find.
  • Develop project management skills it’ll help you relate to PMs.
  • You need to understand the interplay between scope, time, cost and quality.
  • You need to understand how to use assumptions instead of simply making them.
  • Learn some business analysis, coding and application architecture.
  • Become as technical as you need to be.
  • Learn to present and facilitate.
  • Become comfortable at a whiteboard.
  • Learn to read statistics, and to know when you’re being misled. Better yet learn to manipulate data for yourself.
  • Pick up lots of analysis techniques: FMEA, SWOT, force field, the list goes on.

To be concluded.

The Untempered Schism

“He stood in front of the Untempered Schism. It’s a gap in the fabric of reality through which could be seen the whole of the vortex. We stand there, eight years old, staring at the raw power of time and space, just a child. Some would be inspired. Some would run away. And some would go mad.”  Doctor Who, The Sound of Drums (2007).

That is how I felt when first understood context.

Not when I read the words “The value of any practice depends on its context”, and thought Well obviously.

Nor when I started taking context into consideration by tweaking my approach, and thought Hey, I’m context driven.

No, it was when I first took in the variety and beautiful complexity of context, when the world turned under my feet and I realized that the rabbit hole keeps on going, when context became the motivation for my testing.

How do you deal with that? The brain numbing diversity of factors you need to take into account when assessing context? The incalculable number of approaches, techniques and skills that might apply? Beat a retreat? Fall back to the comfort zone? To familiar practices and the words of a favorite textbook? To default behavior and the way we do it around here?

Once you appreciate context, these are not options.

Yet for those who haven’t been exposed to context driven testing, there is frequently ONE big idea, ONE option, ONE approach that has been considered. More often than not, this is the FIRST idea that came to mind. This isn’t context driven, it is laziness driven. At best it is driven by a lack of imagination. Some of the time this one idea hasn’t even been thought through to a point where it is even remotely defensible. In the BBST Test Design lectures, Cem Kaner identifies this as being a case of “premature closure: adopting the first solution that looks likely to work, instead of continuing to ask whether there are better solutions”.

As service providers, we have an obligation to find better ways to provide value through our testing. To test deeper, faster, cheaper, whatever our projects demand. To find solutions to testing problems. A single idea doesn’t cut it.

Exploring context isn’t easy. Nor is imagining and evaluating alternatives, nor adjusting to meet changing demands. There’s no algorithm for this, no simple set of repetitive steps that will lead us inexorably to a solution. There is only skill and judgment.

To be continued.

Even in Context

1. The value of any practice depends on its context.

2. There are good practices in context, but there are no best practices.

So say the first two principles of context driven testing. But are there best practices in context? Are there practices that represent the best or only choice in any given situation?

First we need to discuss context. What is context? That’s a pretty complex question, if only because context is not a single thing. Context is not indivisible, it consists of many factors.

When exploring context, I ask a lot of questions. The following list is far from exhaustive, and is only intended to illustrate the range and diversity of factors that make up context:

  • Who are the participants of the project?
  • Who are the sponsors of the project?
  • Who is paying for the project?
  • Who are the customers of the project?
  • Who are the customers of the testing?
  • Who will use the software being developed?
  • Who else might be affected by the software being developed?
  • Where are the various stakeholders located?
  • What different organizations or organizational units do they belong to?
  • What contractual factors are involved?
  • What political factors are involved?
  • How effectively do the various participants interact?
  • What expectations do the stakeholders have for software quality?
  • What quality characteristics matter to them?
  • What other project constraints (scope, time, cost) matter to them when weighed against quality?
  • What reporting expectations do they have?
  • What methodological, standards or regulatory expectations do they have?
  • How do expectations vary between stakeholders?
  • Which stakeholders really matter?
  • What development methodology are they using?
  • What experiences do the project team have with this methodology?
  • What is the project delivering?
  • When is it expected to deliver?
  • When do people really think it will deliver?
  • What is the release strategy? Is this a one-off or will there be multiple releases?
  • What bits of the solution be given to testing when?
  • Will those bits change often?
  • What information is available?
  • How current is it?
  • How likely is it to change?
  • How accurate is it perceived to be? Will it help or hinder you?
  • What might fail and how?
  • How likely are such failures?
  • Why would a failure matter?
  • What impact would a failure have on the stakeholders?
  • How risk averse are the stakeholders?
  • What keeps them up at night?
  • How complex is the solution?
  • What technology is being used?
  • How established is this technology?
  • How experienced are the developer in using it?
  • How testable is the solution?
  • How likely are the stakeholders to listen if you need to ask for testability enhancements?
  • Who will be doing the testing?
  • Who will be testing which bits?
  • What tools do testers have at their disposal? What do they need?
  • What skills do the testers have?
  • What weaknesses do they have?
  • What influence do the testers have? Are they credible or simply ignored?
  • Why are YOU involved?

It is highly unlikely that any practice will be a perfect fit for the context you are testing in, there are always trade offs involved. For example:

  • A practice might look like a fit in terms of the quality goals of project stakeholders, yet perform poorly in terms of time and cost constraints.
  • A practice might look like a fit in terms of time, cost and quality, but be unworkable when it comes to the capabilities of the test team.
  • A practice might fit perfectly with the capabilities of the testers, yet be wholly inappropriate for revealing useful information about the solution itself.

So the value of any practice in context cannot be measured on a single scale: it will be a blend of different strengths and weakness that play to a multitude of different contextual factors. As if this weren’t challenging enough, different stakeholders will place different values on different factors:

  • One stakeholder might value time and cost over quality.
  • One might be terrified of a lawsuit and want a LOT of testing.
  • One might have an expectation that testing standards be used, and believe that time and cost should be adjusted accordingly.

In the same way that “Quality is value to some person” (Weinberg), so the value of a practice in context is its value to some person. Quality is subjective, no less so the quality of testing practices. An individual stakeholder might perceive that a given practice is best – for them – in context, but there are no absolutes that will apply to all stakeholders.

There are no best practices, even in context.


Hard Sometimes

Being context driven can be hard sometimes.

It is easy to get carried away with a favoured approach, a preferred method of doing things, a practice that you’ve used successfully before.

This is perfectly normal: when we start out on a new project, we’re juggling fifty million new pieces of information, struggling to make sense of it all and figuring out where to start. The filter of our experience helps us to simplify these problems rather than having to sweat through everything from first principles. If we did the latter we’d be paralyzed, like a deer in the headlights. Sometimes this is less benign: sometimes it is easy to get carried away with a new shiny thing, context be damned. I find that the lure of clever and elegant automation solutions can be particularly hard to resist.

But we’re not testing for kicks. We’re not testing because we want to do things the way we like doing them. We’re not testing because we want to build a cool automation gizmo. We have a mission: we’re testing to provide a service to our projects by doing some violence to their software. Many of our choices demand closer attention.

So think: are you considering an approach because it is a good fit, or because it looks fun or because you’re operating on instinct?

So be honest: recognize and acknowledge your biases, admit them to others.

So be open to challenge: invite others to suggest alternatives or find reasons why your own way might not work.

So compare and contrast: explore the similarities and differences between approaches. How does each serve your mission? How do they stack up against any constraints you may be under?

So consider history: explore the similarities and differences between situations where you’ve used an approach before, and the current context. This can reveal reasons a previously successful approach might fail, or a previously unsuccessful approach might just pay off.

Being context driven can be hard sometimes. Remembering to think is a good first step.

Mission Creep

“Testing was going so well” said the tester, “at least on the first release”.

“We had a clear mandate: find as many important bugs as quickly as we could, and report them to the developers. And we found lots: the PM was happy because we were giving her a good feel for quality, the developers were happy because we were giving them rapid feedback, and the test team was happy because we felt like we were doing our jobs well.”

“I suppose it was during acceptance testing that things started to change. The UAT team hadn’t had enough exposure to the app during development, and struggled to figure out how to run their tests. In the end, the PM asked us to help out. We were only too happy to: we starting designing and maintaining scripted walkthroughs of most of the key features and requirements, as well as authoring the associated test data. This took a fair amount of effort, but we were up for it: it’s a team effort after all.”

“The initial release went in pretty smoothly, I mean, some bugs crept through, but we helped out there too: a lot of what support were getting hit with, we were able to find workarounds for, anything else we were at least able to repro and isolate for the devs. We still do a lot of that now: it helps to keep up a good relationship with the support team.”

“The latest release was a lot hairier; a fair few of the devs have changed. The new guys struggled to understand why a lot of the unit tests were failing, and ended up commenting them out: this meant we started seeing a lot more regression bugs. Added to that, they’re offshore: now we’ve got developers on three continents. Communications don’t seem to be hanging together and somewhere along the line config management got messed up. We ended up doing a lot more regression testing this time around.”

“Got to go, post mortem starts in five minutes.  Release 2 went in last week, and the PM is on the war path: she can’t understand how we missed so many major bugs.”

What happened here?

In the story above, the testers started out with a clear mission: find bugs.

…then they began to provide scripted demonstrations for acceptance testing.

…then they started to figure out workarounds and do isolation for the support team.

…then they added black box regression tests to mitigate regression risks.

…and then they started to fail in their initial mission.

After their initial success, they allowed their mission to expand beyond its original goals, and fell foul of mission creep.

Mission creep brings a number of risks:

  • Loss of effectiveness. There are many possible missions for testing, for example: finding bugs, investigating quality, reducing support costs, mitigating risks, reducing liability, conforming with standards or regulations1. Whilst any of these is potentially valid, some practices are more suitable for some missions than others. If changes are not recognized, and practices not changed accordingly, a test team can find itself working in a way that runs counter to its mission.
  • Loss of focus. Different goals can be blended, but this requires practices to be blended too. This adds complexity. If you have try to do too many things, you may not be able to do any of them well.
  • Overextension. Like its project management cousin scope creep, increasing the number of goals often requires additional effort. Without a corresponding increase to time or resources, mission creep means that less effort can be allocated to each goal, making success of any goal all the less likely.

How can you address mission creep?  Here are some suggestions:

  • Make the mission explicit. Consult with your stakeholders and ensure that the mission is both understood and agreed. If appropriate (for example: if you are an external contractor or testing vendor), consider making the mission a part of your formal scope.
  • Keep an eye open for your mission changing or expanding. Don’t let change surprise you. Review regularly, and engage stakeholders in this process. Are all needs being satisfied?  Does the project need anything new out of testing?
  • Make adjustments to mission and supporting practices. Don’t let your mission and practices diverge, but consult with your stakeholders about the tradeoffs. What are the time and cost implications? Do any goals conflict? Consider whether it is possible to refocus: are any goals now redundant?

Testing missions evolve. Unrecognised, mission creep can cause significant damage to your ability to deliver what your project needs and expects of you. If you are testing the same software for any length of time, mission creep is almost inevitable: it needs to be managed.


See Lessons Learned in Software Testing (Kaner, Bach & Pettichord) for a discussion of missions in testing.