Tag Archives: Learning

The Validity of the Testing Environment

From art experts recognizing a statue as a fake (Gladwell, 2005) to nurses recognizing obscure signs of infection in infants (Klein, 1998), the human mind is capable of performing amazing feats of intuition. As magical as it might appear, intuition is little more than recognition of a situation (Simon, 1992). As expertise develops, the decision maker is able to recognize an increasing array of contexts and match an appropriate response from a repertoire of possible actions. It is also subject to bias, systematic failures of judgment that the decision maker can walk blindly into. Intuition, it seems, is not accompanied by any subjective signal that marks it as valid, and confidence is not a reliable indicator as to the validity of intuition (Kahneman & Klein, 2009).

Daniel Kahneman and Gary Klein point out that professional environments differ in terms of whether they are conducive to the development of intuitive expertise (Kahneman & Klein, 2009). Whilst firefighters and nurses (amongst others) can develop such expertise, stock-pickers and political pundits do not seem to be able to do so. In the latter case, we would be better served to place our trust in algorithms and other such rational methods than to rely on the advice of experts. What about testers? Can we develop intuitive expertise? Or would we be better off relying on process, decision criteria and other mechanisms?  Given the controversy over best practices, and the importance that context driven testers place on judgment and skill, this would seem to be a fundamental question.

Kahneman and Klein identify two factors that are necessary conditions for the development of expertise: that the environment provides valid contextual cues, and that it provides adequate opportunities to learn through timely and relevant feedback. So how do the environments that we test within stack up against these criteria?

Firstly, let’s look at contextual cues. Does testing provide the kind of cues that facilitate learning? Given that every context is different, that we are not repeatedly producing the same software, one might be forgiven for believing that any such cues provide weak statistical signals. A comparison might be helpful. Consider chess: both chess and testing are intractable problems. It is widely acknowledged that it is impossible to test everything; similarly, the average chess game has in the order of 10120 possible positions. Whilst this would be unfeasible to brute force, grand masters are able to immediately recognize anywhere between fifty and a hundred thousand patterns and select a strong move within a matter of seconds. In short, chess is a paragon for expertise: it provides a valid environment despite the fact that individual cues might be presented only infrequently. We should not mistake complexity for invalidity: the validity of an environment is not solely determined by the frequency with which individual cues occur, but also by the relevance of those cues. For the tester who is willing to listen, there is an abundance of such cues: the values of customer and stakeholder, the interplay between project goals and constraints, the makeup of technical solution. I’ll discuss contextual cues again in the near future, both at CAST 2012 and in an accompanying post.

Expertise is not experience. Without the opportunity to learn, all the experience in the world will not lead to expertise. Let’s turn to that. The learning opportunities present in any given environment are determined by the quality and speed of the feedback that it provides. In testing, this varies. In some cases we are often able to gain feedback, such as our stakeholder’s reactions to our approach, strategies and the bugs that we find. In other cases it can be difficult to get rapid and relevant feedback, for example: on the bugs that we miss. Sometimes these stay missed for a long time, whilst in some contexts we get feedback of the wrong kind and risk learning the wrong lessons. For example if feedback takes the form of a witch hunt, an attempt to allocate blame, just what does that teach testers? Even where this is avoided, we often see an emphasis on processes and methods and how they might be improved, rather than a focus on what individual testers might learn from their mistakes. Perhaps an environment in which human judgment has been surrendered to process is one in which the conditions for the development of expertise have been removed. Not only are factory approaches blinkered and dehumanizing, but they might well rob testers of their means of escape: an opportunity to develop expertise. There are however some intriguing possibilities. Could we train ourselves to become better observers so as to be more receptive to the feedback that our environments supply? Is it possible to reconfigure our environments such that they provide better feedback? Dedicated testers can improve their chances of developing expertise by creating and nurturing feedback loops.

Perhaps asking whether the testing environment is conducive to developing expertise is too simplistic a question. Kahneman and Klein identify some professions as fractionated: where expertise is displayed in some activities but not in others. Given the uneven nature of feedback, it may well be that testing is one such profession: there may be some activities in where it is more appropriate to draw on algorithmic methods than on expertise. Of course, the trick is recognizing when to do so, recognizing the limits of our expertise. And that requires judgment: as James Bach tweeted at the weekend “In any matter that requires sapience, even algorithmic methods can only be applied heuristically.



  • Gladwell, M. (2005). Blink.
  • Kahneman, D. and Klein, G. (2009). Conditions for Intuitive Expertise: A Failure to Disagree.
  • Klein, G. (1999). Sources of Power: How People Make Decisions.
  • Simon, H. (1992). What is an Explanation of Behavior?

The Heart of Testing

There are many definitions of testing, some excellent, some banal. Most hinge on its purpose, its deliverable or its activities, yet such descriptions feel inadequate somehow, as if providing only a fleeting glimpse into our craft. They fail to tell the story of what is going on at the heart of testing.

Testing, I believe, is about learning.

To begin with, there is learning about the software. What does it do? How does it behave in a myriad of different situations? As Pete Walen recently remarked: “Testing is neither confirmatory nor destructive.  Instead, testing is an interrogative activity that allows us to discover the true behavior of the software”. I couldn’t agree more, though this is perhaps the most obvious of the areas in which we must learn. Discovery is only part of testing. Try this out: stop reporting bugs, start reporting the results of your every observation as to how software reacts to varying stimuli, enumerate each case in painstaking detail. Any value that you might bring will be lost in the noise.

Testing is not merely a technical investigation, it requires judgment concerning the ways in which software adds to or detracts from value. And how do we do that? Value is relative, it is subjective. To understand the value of software we must first learn about the values of our stakeholders. What does “quality” look like to them? What things are important, what do they hold dear? What might be an unacceptable compromise, to what are they indifferent?  Together, our discoveries about the software and about our stakeholders enable us to draw inferences, to make judgments, as to the value that software provides. Together, perhaps, these two aspects of learning provide a useful portrayal of testing, yet to be a successful tester there are many more aspects of learning. We must be capable of learning at many different levels besides.

How do we learn about software? How do we learn where to look, or more accurately where to look to find the problems that will matter most? How do we actually learn about the values of our stakeholders? How do we learn the levers to pull in order to establish credibility or legitimacy? To learn these things, we much learn something else: we must learn how to interact with our stakeholders, regardless of how technical they are, regardless of how articulate they are, regardless of how time pressured they are, regardless of whether they give a damn about the finer points of testing. Every project has its own culture, its unwritten rules, its norms and power relationships. Every project will be different: there is no process, methodology or communications plan that will tell you how to do this. You need to immerse yourself in the project and learn how to interact, much as an immigrant needs to learn how to interact with natives of their host culture.

And if this were not enough, the range of skills and knowledge that a tester should, or could, apply is vast beyond comprehension. We constantly find ourselves learning new skills, even inventing or reinventing them. No amount of learning can be considered enough, so we must become experts at learning, and this requires that we learn about learning itself. How do we quickly acquire and assimilate new information and skills? How do we apply them? How do we choose what to learn next?

Learning lies at the heart of testing. Viva the learning tester.

The Long Way Round

At high school, I learned remarkably little: or to be more precise, I remember very little of what I was taught. As best as I can recall, my teachers took me for a whirlwind tour of their syllabi. Facts flew by, each another faceless town; there was never any opportunity to stop and look around. Taking computer science as an example, whilst I struggle to remember a single thing we did in class the lessons that I learned whilst writing keyloggers and DoSing the Vax have stuck with me to this day.

As an undergraduate, I was…well, let’s just say that I was a little distracted. I remember pieces of the syllabus, but debt and hunger taught me the most important lesson of those years: they taught me a work ethic. As a postgraduate, things were different. The opportunity costs of taking time out from work were too great, so I studied for my masters in my own time. Many facts continued to pass me by, sticking long enough to write an exam, but each term I’d learn new ways of looking at a subject, and find things that resonated with my daily working experiences. I learned about ways of solving problems that I could put into practice immediately. I developed skills. I began to learn how to learn: I realized that facts don’t stick too well, but that skills and experiences do. And in an age where a dazzling array of facts is just a few clicks away, what is the value of memorizing an encyclopedia?

When I started testing, I muddled through for a while until a hideously embarrassing interview showed me the extent of my own ignorance. I resolved to learn more and applied what I’d learned as a postgrad. I read, I studied, and I followed many interesting diversions. I applied anything and everything I could on the projects I was testing on. I learned SQL when I needed to. I taught myself Java when I needed some quick and dirty automation. I sought opportunities to teach when I realized the power that this has to deepen my own understanding. I also learned that the toughest most painful projects are invariably those that provide the greatest opportunities for growth.

This week, I was fortunate enough to attend Rapid Software Testing (RST) run by Michael Bolton. This is a splendid course that focuses on skills, mindset and different ways of breaking down testing problems. It does so through experiences: games and exercises that are fun and sometimes a little painful too. I merrily leapt feet first into a number of traps; and suitably humbled found both new ideas and added depth to old knowledge. Whilst there is no silver bullet when it comes to learning, I have no doubt that I will continue to draw lessons from RST for years to come. Learning is a lifelong journey, not a whistle stop tour. When there’s no destination, then short cuts make no sense. I’m enjoying taking the long way round.

Learning for the Long Haul, Part 2: Touring & SBTM

“Tell me and I’ll forget; show me and I may remember; involve me and I’ll understand.” Chinese Proverb.

In the previous post I introduced the problem of building and maintaining knowledge (primarily application knowledge) within a testing team. I then described some simple artefacts that could serve as cost-effective (and opportunity-cost effective) alternatives to “idiot-scripts”. Such artefacts may prove useful as a means to introduce new testers to a given piece of software, but are a starting point at best. The bottom line: much of what we learn, we learn by doing. To effectively build knowledge we need more than passive methods (reading a document, browsing a model, watching a video), we need ways in which new testers can become actively engaged in their own learning.

Over the last few years, I have employed a handful of different approaches to doing so. This has included setting dedicated “play time” during which the new testers freely roam the software, and secondments where testers move into a new team and gradually build up their knowledge through testing on a live project. Whilst the results have been adequate, there is room for improvement:

  • In many cases testers were able to pick up the kind of information that was expected from them; in some cases this learning simply took too long.
  • The use of “play time” had mixed results: some testers were able to pick up significant amounts of information about the software whereas others floundered in the absence of specific learning objectives.
  • Assessment of progress was difficult, highly subjective, and not particularly granular (“Can Fred test that now?”).

As a result, I’ve started looking for ways in which this kind of learning can be enhanced such that testers:

  • Are active participants in their own learning.
  • Have specific learning objectives.
  • Are provided with a structure that supports goal setting, reflection and feedback.

In order to address these points, I am looking at a blend of tours and session based test management (SBTM).

First, let’s discuss tours. Tours are a set of simple techniques for getting familiar with a software product: they are often used within exploratory testing to emphasise the “learning” aspect of the ET triumvirate (learning, design and execution). Michael Kelly provides a good overview of a number of different types of tour in Taking a Tour Through Test Country.

Using this approach, a tester explores an item of software in order to learn about it. Some example missions are listed below:

  • Identify and list the software’s features.
  • Identify and list the software’s variables, include any thoughts as to equivalence classes and boundaries.
  • Identify and list the software’s data objects.
  • For a given data object, create a state diagram that represents its life-cycle.
  • Create a map describing how the user navigates the software.
  • Identify and list any potential or claimed benefits of the software.
  • Identify and list any decision rules implemented by the software, for complex sets of rules representing these as a decision tables or cause-effect graphs.
  • Identify and list different ways in which the software can be configured, and the consequences of each configuration.
  • Identify ways in which the software interacts with other systems with which it interfaces: draw a sequence diagram.

This is far from exhaustive, and Kelly’s article includes a number of other ideas.

In these examples, I’ve coupled learning objectives with specific deliverables such as the creation of models or inventories. I used a similar approach on a project where I was using ET with the goal of reverse engineering a product rather than testing it: in doing so I found that creating a model whilst I explored provided additional focus, helped me to keep track of my progress and helped in identifying additional avenues that might be worth investigating (at the time, I likened this to keeping a map whilst playing a 1980’s text based computer game). An added benefit in the context of learning is that such models can serve as an assessable deliverable (more on this below).

Now to SBTM: Jon Bach describes Session Based Test Management as a means to organize exploratory testing without obstructing its flexibility. This seems a reasonable fit: not only is it a process with which many testers are familiar, but it also enables goal setting, reflection and feedback within a structure that is flexible enough to adapt quickly to a tester’s individual learning needs.

As I am using this to structure learning rather than testing, I’ve made a few tweaks. Here’s an overview:

The general idea is that the tester’s learning is broken into manageable sessions, each with a specific learning mission. This mission is agreed by the tester and his coach (another team member with more experience of this particular software).

With the mission established, the tester is free to tour the software or any associated material with the goal of fulfilling that mission. Whilst doing so, he constructs any agreed deliverables.

On completion of the session, the tester creates a short report that outlines what was achieved. This gives the tester an opportunity to reflect on what he has learned, how any new learning relates to his previous knowledge about the software, and what else could be investigated.

Finally, the coach and tester perform a debrief in which the session report and any deliverables are reviewed. This gives the tester an opportunity to further refine his thoughts so as to be able to articulate his learning to his coach, whilst allowing the coach to assess what the tester has learned and provide any feedback that she feels is appropriate. This debrief is also an opportunity for the coach and tester to agree potential follow-up sessions, allowing them to tailor the route through any application-specific curriculum to the needs of the individual tester.

This is a work in progress, and I’ll write more on this once I’ve tested it out further. In the meantime, if you have any thoughts or ideas on the subject – or have attempted anything similar – I’d love to hear from you.

My thanks to Becky Fiedler: who helped me to coalesce some of these thoughts, whilst adding to my reading list 🙂

Learning for the Long Haul, Part 1: Onboarding

Scenario 1
Project Manager: Hey, sorry about the short notice, but we’ve got a new build coming down the pipe. We really need Bob on this one next week, he’s the expert.
Test Manager: Er, sorry, Bob’s an in-demand guy…I’ve got him allocated to XYZ for the next month, no one knows it like him.
Project Manager: [disappointed silence].

Scenario 2
Tester: You know, I really like the sound of that role on the XYZ project, it’s the kind of testing we talked about when we discussed my development plan – just what I’ve been trying to get into.
Test Manager: Ugh, well…I’m going to have to think about that. It’d leave your current team short-handed, and I don’t really have anyone else with the application knowledge to backfill you right now…
Tester: [sound of resume being updated].

Scenario 3
Test Manager: Great news!  I know your team is short-handed right now, I’ve got a new tester for you.
Test Lead: Great. With our timescales, how do you think I’m going to get a newbie trained up?  That’s going to reduce our capacity not increase it.
Test Manager: [mutter].

These scenarios might sound familiar. What they have in common is that they all suggest problems in how knowledge is being managed.

Knowledge is the tester’s stock in trade. It is from a position of knowledge that we construct our models as to how software should work, and how it could fail. Testing itself can be seen as a process whereby we refine these models, reducing uncertainty as we accumulate more knowledge. Within my context, as a manager of teams who provide ongoing testing services for multiple applications, effective management of knowledge is critical. Failure to do so results in the concentration of expertise in too small a group of individuals. This can give rise to a number of risks and issues:

  • Teams lack flexibility when it comes to assigning people to different projects, the result of which is often ineffective resource allocation: some teams are left short-handed, others lack the knowledge they require.
  • Team capability is placed at risk should a critical member leave.
  • Team members may miss out on opportunities to further their careers if their knowledge locks them in to a particular role.
  • Introducing new team members into the mix can be both time consuming and distracting.

A common knee-jerk reaction to these problems is idiot-level scripting, i.e. script to such a level of detail that “any idiot off the street” could run the tests (just imagine how good that testing would be). Unfortunately, scripts are of limited value as a vehicle for learning and the development and maintenance of such scripts represents a significant opportunity cost.

When it comes to providing information to new team members, there are alternatives. Here are a few that some of my teams have successfully used:

  • Build models. A picture is worth a thousand words, or in this case a thousand scripts. Models are an exceptionally powerful ways to convey information, and are far more efficient at doing so than scripts.  Not only can they describe how software should work, but they can also suggest tests. Models can easily be harvested from test basis documents, or created on the fly as testers gain a better understanding of a given item of software. Examples might include feature inventories, use cases, decision tables, state diagrams, context diagrams and so on.
  • Create “how to” guides. Perhaps certain tasks will be performed frequently during testing.  Why not borrow an automation pattern and decouple the scripts from the tests? Maintaining a set of instructions in one place is far more efficient than replicating it in hundreds or thousands of places, and once testers have become familiar with a given procedure they are unlikely to rely on the instructions any more – giving them back their peripheral vision.
  • Make screen cam or video walkthroughs. A cheap alternative to the above, a quick screen cam of a given task is easy to produce, leaves less margin for error when it comes to different interpretations of instructions, and can include voice-overs that provide lots of additional information.

Of course, none of the above are a substitute for experience: if you’ve never ridden bicycle before, try reading the manual first then see how well you do.  What the above approaches do provide are ways in which learning can be facilitated without the opportunity cost of idiot-scripts. The rest is down to testing.

I’ll discuss strategies for building the knowledge of testers, and propagating knowledge throughout a test team, in later posts.

Two Tales of Learning

Riding a bicycle is a complex skill to acquire: not only must a rider stay balanced, but also steer, pedal, remain aware of his or her surroundings and even brake all at the same time.

Enter stabilisers (training wheels). By taking balance out of the equation, these reduce the complexity a little, and let the new rider focus on learning and coordinating the other skills.

When my daughter first learned to ride her bike, I assumed that once she grew confident with stabilisers, it would be easy enough for her to ride without them. Along came the big day, off came the extra set of wheels, and…disaster…over she went. A fluke accident? Once the tears had dried, she reluctantly agreed to try again…to the same result.

But why did she have this difficulty, once the stabilizers had come off? Simply put, she’d mastered the other aspects of riding, but had relied on her stabilisers, never learning to balance on her own.

More recently, I experienced something similar. Last year I bought a GPS for the car. I’ve always prided myself in both my direction sense and my map reading skills, so had always avoided this particular gadget. However, facing the prospect of a 900 mile drive to New York with my family, I decided it would be wise not to rely solely on a map (and to this day believe that the use of this little device saved my marriage).

Having been quite impressed with the GPS, I started using it more frequently. When I was called upon to visit our office on Prince Edward Island for the first time, out came the GPS. Over the next few months, I made several trips to the same location, each time using the GPS. Then one day…disaster…dead battery, no charger and no map in the car. It took me several wrong turns, and many miles of exploring before I finally figured out my route.

What had happened? Pre-GPS I would have used a map and a few scribbled notes on my first journey, then would easily have been able to find my way on each subsequent trip. Why was it different this time? I had been relying on the GPS, much in the same way that my daughter had relied on her stabilisers. I had relied on its detailed turn-by-turn instructions, and what’s more, had focussed on following those instructions to such a degree that I hadn’t even taken in my surroundings.

These are both true stories, but what have they got to do with testing?

In Scripting: An Industry Worst Practice?, Kaner and Bach use the navigation analogy as part of a refutation of the assumption that test scripts make effective training wheels. This, and the assertion that scripts make inexperienced testers productive whilst they are learning, are both frequently cited arguments in favour of scripts.

Can a script help a new tester learn? As in the story about my daughter, perhaps by removing some of the variables from the testing equation the tester can focus on learning some aspects of their role. This might include gaining an overview of the software’s features, its basic operation, and the common checks that are frequently performed during testing. Equally, reliance on scripts will prevent the tester from learning other things. How much does a script teach about the needs of users, of the rationale for a particular piece of software, the underlying rules that it implements, or even how to test it such that important failures will be revealed? A test script abstracts these things away in the same way that stabilisers take the balance out of riding a bike. Scripts are a limited learning tool.

Can a script help a new tester test productively? As with my GPS, perhaps a script will get a tester to their destination. This might be the end of the script, or to a point that they identify that the script cannot be completed due to bugs. This might add value to the testing effort if simple checking is called for, for example when mitigating regression risks (although such an approach has issues of its own, as discussed in this series of posts). However, testing in such a manner checks only the things that the script was designed to check for: tangentially related bugs that would be obvious to anyone actually using the software are likely to be missed because the tester will be focused on following instructions and not on taking in their surroundings – in exactly the same way that I wasn’t taking in my surroundings when using my GPS. This is a dangerous approach and a good way to miss bugs.

Executing scripts can offer some limited value. However, for either learning or testing to be truly effective a healthy dose of trial and error is required. Several weeks after the stabiliser came off, and after a few more spills, my daughter was finally riding with confidence. I no longer use my GPS for trips to Prince Edward Island, and since exploring some of the back roads of Nova Scotia, I have never lost my way again.

The Parable of the Black Belt

Picture a martial artist kneeling before the master sensei in a ceremony to receive a hard-earned black belt. After years of relentless training, the student has finally reached a pinnacle of achievement in the discipline.
“Before granting the belt, you must pass one more test,” says the sensei.
“I am ready,” responds the student, expecting perhaps one final round of sparring.
“You must answer the essential question: What is the true meaning of the black belt?”
“The end of my journey,” says the student. “A well-deserved reward for all my hard work.”
The sensei waits for more. Clearly, he is not satisfied. Finally, the sensei speaks. “You are not yet ready for the black belt. Return in one year.”
A year later, the student kneels again in front of the sensei.
“What is the true meaning of the black belt?” asks the sensei.
“A symbol of distinction and the highest achievement in our art,” says the student.
The sensei says nothing for many minutes, waiting. Clearly, he is not satisfied. Finally, he speaks. “You are still not ready for the black belt. Return in one year.”
A year later, the student kneels once again in front of the sensei. And again the sensei asks: “What is the true meaning of the black belt?”
“The black belt represents the beginning — the start of a never-ending journey of discipline, work, and the pursuit of an ever-higher standard,” says the student.
“Yes. You are now ready to receive the black belt and begin your work.”

— from Built to Last: Successful Habits of Visionary Companies (James C. Collins and Jerry I. Porras.)

When I was new to testing, I thought I understood it. Back then testing looked pretty simple: write some tests, run them, report the results.

A few years on, I thought that perhaps I was getting close to mastering it, even if things looked a little more involved: many types of testing, different strategies and approaches, a variety of test design techniques, a number of process options etc. I made it my mission to study, and attempt to practice, each one.

Now, I accept the fact that the learning is without end: the more you knows about the subject, the more you realize is still unknown. The parable of the black belt reminds us that each milestone along the way is in fact just the beginning of our next stage of development.

And thank goodness for that! Imagine being in a state of total mastery, of total knowledge. That looks like a pretty desolate place to me: without the thrill of discovery, and with no little surprises to look forward to.

Instead, it is reassuring to embrace this truth and face each day with the humility required to keep learning.