“Tell me and I’ll forget; show me and I may remember; involve me and I’ll understand.” Chinese Proverb.

In the previous post I introduced the problem of building and maintaining knowledge (primarily application knowledge) within a testing team. I then described some simple artefacts that could serve as cost-effective (and opportunity-cost effective) alternatives to “idiot-scripts”. Such artefacts may prove useful as a means to introduce new testers to a given piece of software, but are a starting point at best. The bottom line: much of what we learn, we learn by doing. To effectively build knowledge we need more than passive methods (reading a document, browsing a model, watching a video), we need ways in which new testers can become actively engaged in their own learning.

Over the last few years, I have employed a handful of different approaches to doing so. This has included setting dedicated “play time” during which the new testers freely roam the software, and secondments where testers move into a new team and gradually build up their knowledge through testing on a live project. Whilst the results have been adequate, there is room for improvement:

• In many cases testers were able to pick up the kind of information that was expected from them; in some cases this learning simply took too long.
• The use of “play time” had mixed results: some testers were able to pick up significant amounts of information about the software whereas others floundered in the absence of specific learning objectives.
• Assessment of progress was difficult, highly subjective, and not particularly granular (“Can Fred test that now?”).

As a result, I’ve started looking for ways in which this kind of learning can be enhanced such that testers:

• Are active participants in their own learning.
• Have specific learning objectives.
• Are provided with a structure that supports goal setting, reflection and feedback.

In order to address these points, I am looking at a blend of tours and session based test management (SBTM).

First, let’s discuss tours. Tours are a set of simple techniques for getting familiar with a software product: they are often used within exploratory testing to emphasise the “learning” aspect of the ET triumvirate (learning, design and execution). Michael Kelly provides a good overview of a number of different types of tour in Taking a Tour Through Test Country.

Using this approach, a tester explores an item of software in order to learn about it. Some example missions are listed below:

• Identify and list the software’s features.
• Identify and list the software’s variables, include any thoughts as to equivalence classes and boundaries.
• Identify and list the software’s data objects.
• For a given data object, create a state diagram that represents its life-cycle.
• Create a map describing how the user navigates the software.
• Identify and list any potential or claimed benefits of the software.
• Identify and list any decision rules implemented by the software, for complex sets of rules representing these as a decision tables or cause-effect graphs.
• Identify and list different ways in which the software can be configured, and the consequences of each configuration.
• Identify ways in which the software interacts with other systems with which it interfaces: draw a sequence diagram.

This is far from exhaustive, and Kelly’s article includes a number of other ideas.

In these examples, I’ve coupled learning objectives with specific deliverables such as the creation of models or inventories. I used a similar approach on a project where I was using ET with the goal of reverse engineering a product rather than testing it: in doing so I found that creating a model whilst I explored provided additional focus, helped me to keep track of my progress and helped in identifying additional avenues that might be worth investigating (at the time, I likened this to keeping a map whilst playing a 1980′s text based computer game). An added benefit in the context of learning is that such models can serve as an assessable deliverable (more on this below).

Now to SBTM: Jon Bach describes Session Based Test Management as a means to organize exploratory testing without obstructing its flexibility. This seems a reasonable fit: not only is it a process with which many testers are familiar, but it also enables goal setting, reflection and feedback within a structure that is flexible enough to adapt quickly to a tester’s individual learning needs.

As I am using this to structure learning rather than testing, I’ve made a few tweaks. Here’s an overview:

The general idea is that the tester’s learning is broken into manageable sessions, each with a specific learning mission. This mission is agreed by the tester and his coach (another team member with more experience of this particular software).

With the mission established, the tester is free to tour the software or any associated material with the goal of fulfilling that mission. Whilst doing so, he constructs any agreed deliverables.

On completion of the session, the tester creates a short report that outlines what was achieved. This gives the tester an opportunity to reflect on what he has learned, how any new learning relates to his previous knowledge about the software, and what else could be investigated.

Finally, the coach and tester perform a debrief in which the session report and any deliverables are reviewed. This gives the tester an opportunity to further refine his thoughts so as to be able to articulate his learning to his coach, whilst allowing the coach to assess what the tester has learned and provide any feedback that she feels is appropriate. This debrief is also an opportunity for the coach and tester to agree potential follow-up sessions, allowing them to tailor the route through any application-specific curriculum to the needs of the individual tester.

This is a work in progress, and I’ll write more on this once I’ve tested it out further. In the meantime, if you have any thoughts or ideas on the subject – or have attempted anything similar – I’d love to hear from you.

My thanks to Becky Fiedler: who helped me to coalesce some of these thoughts, whilst adding to my reading list

Scenario 1
Project Manager: Hey, sorry about the short notice, but we’ve got a new build coming down the pipe. We really need Bob on this one next week, he’s the expert.
Test Manager: Er, sorry, Bob’s an in-demand guy…I’ve got him allocated to XYZ for the next month, no one knows it like him.
Project Manager: [disappointed silence].

Scenario 2
Tester: You know, I really like the sound of that role on the XYZ project, it’s the kind of testing we talked about when we discussed my development plan – just what I’ve been trying to get into.
Test Manager: Ugh, well…I’m going to have to think about that. It’d leave your current team short-handed, and I don’t really have anyone else with the application knowledge to backfill you right now…
Tester: [sound of resume being updated].

Scenario 3
Test Manager: Great news!  I know your team is short-handed right now, I’ve got a new tester for you.
Test Lead: Great. With our timescales, how do you think I’m going to get a newbie trained up?  That’s going to reduce our capacity not increase it.
Test Manager: [mutter].

These scenarios might sound familiar. What they have in common is that they all suggest problems in how knowledge is being managed.

Knowledge is the tester’s stock in trade. It is from a position of knowledge that we construct our models as to how software should work, and how it could fail. Testing itself can be seen as a process whereby we refine these models, reducing uncertainty as we accumulate more knowledge. Within my context, as a manager of teams who provide ongoing testing services for multiple applications, effective management of knowledge is critical. Failure to do so results in the concentration of expertise in too small a group of individuals. This can give rise to a number of risks and issues:

• Teams lack flexibility when it comes to assigning people to different projects, the result of which is often ineffective resource allocation: some teams are left short-handed, others lack the knowledge they require.
• Team capability is placed at risk should a critical member leave.
• Team members may miss out on opportunities to further their careers if their knowledge locks them in to a particular role.
• Introducing new team members into the mix can be both time consuming and distracting.

A common knee-jerk reaction to these problems is idiot-level scripting, i.e. script to such a level of detail that “any idiot off the street” could run the tests (just imagine how good that testing would be). Unfortunately, scripts are of limited value as a vehicle for learning and the development and maintenance of such scripts represents a significant opportunity cost.

When it comes to providing information to new team members, there are alternatives. Here are a few that some of my teams have successfully used:

• Build models. A picture is worth a thousand words, or in this case a thousand scripts. Models are an exceptionally powerful ways to convey information, and are far more efficient at doing so than scripts.  Not only can they describe how software should work, but they can also suggest tests. Models can easily be harvested from test basis documents, or created on the fly as testers gain a better understanding of a given item of software. Examples might include feature inventories, use cases, decision tables, state diagrams, context diagrams and so on.
• Create “how to” guides. Perhaps certain tasks will be performed frequently during testing.  Why not borrow an automation pattern and decouple the scripts from the tests? Maintaining a set of instructions in one place is far more efficient than replicating it in hundreds or thousands of places, and once testers have become familiar with a given procedure they are unlikely to rely on the instructions any more - giving them back their peripheral vision.
• Make screen cam or video walkthroughs. A cheap alternative to the above, a quick screen cam of a given task is easy to produce, leaves less margin for error when it comes to different interpretations of instructions, and can include voice-overs that provide lots of additional information.

Of course, none of the above are a substitute for experience: if you’ve never ridden bicycle before, try reading the manual first then see how well you do.  What the above approaches do provide are ways in which learning can be facilitated without the opportunity cost of idiot-scripts. The rest is down to testing.

I’ll discuss strategies for building the knowledge of testers, and propagating knowledge throughout a test team, in later posts.

Picture a martial artist kneeling before the master sensei in a ceremony to receive a hard-earned black belt. After years of relentless training, the student has finally reached a pinnacle of achievement in the discipline.
“Before granting the belt, you must pass one more test,” says the sensei.
“I am ready,” responds the student, expecting perhaps one final round of sparring.
“You must answer the essential question: What is the true meaning of the black belt?”
“The end of my journey,” says the student. “A well-deserved reward for all my hard work.”
The sensei waits for more. Clearly, he is not satisfied. Finally, the sensei speaks. “You are not yet ready for the black belt. Return in one year.”
A year later, the student kneels again in front of the sensei.
“What is the true meaning of the black belt?” asks the sensei.
“A symbol of distinction and the highest achievement in our art,” says the student.
The sensei says nothing for many minutes, waiting. Clearly, he is not satisfied. Finally, he speaks. “You are still not ready for the black belt. Return in one year.”
A year later, the student kneels once again in front of the sensei. And again the sensei asks: “What is the true meaning of the black belt?”
“The black belt represents the beginning — the start of a never-ending journey of discipline, work, and the pursuit of an ever-higher standard,” says the student.
“Yes. You are now ready to receive the black belt and begin your work.”
– from Built to Last: Successful Habits of Visionary Companies (James C. Collins and Jerry I. Porras.)

When I was new to testing, I thought I understood it. Back then testing looked pretty simple: write some tests, run them, report the results.

A few years on, I thought that perhaps I was getting close to mastering it, even if things looked a little more involved: many types of testing, different strategies and approaches, a variety of test design techniques, a number of process options etc. I made it my mission to study, and attempt to practice, each one.

Now, I accept the fact that the learning is without end: the more you knows about the subject, the more you realize is still unknown. The parable of the black belt reminds us that each milestone along the way is in fact just the beginning of our next stage of development.

And thank goodness for that! Imagine being in a state of totally mastery, of total knowledge. That looks like a pretty desolate place to me: without the thrill of discovery, and with no little surprises to look forward to.

Instead, it is reassuring to embrace this truth and face each day with the humility required to keep learning.