Devil in the Detail II

Posted on by
5

Previously…

Continuing with the theme of review and approval as arguments for detailed test scripts…

Argument 2: “We need scripts so that they can be approved prior to execution”.

Like the previous argument, this expresses both a desire and an assumption:

  • DESIRE: We want tests to be approved before they are executed.
  • ASSUMPTION: We can only do that if there are scripts.

Again, we should question if the assumption is valid. It’s much easier for an approver to work through a high level outline of tests than a thousand pages of scripts. I’ve seen approvers asked to do the latter on a number of occasions: I’ve seldom seen an approval result from it.

We should also question the value of such approval. How valid is the approval of tests before they are executed? The tests will change, they must change otherwise many of the insights gained through testing go to waste. As we test, we learn about the software and how it will be used; we learn things that were never considered when the requirements were formulated; we learn implications of the design that were not – could not – be foreseen prior to it being materialized. Further, our learning and observations cause change: as we identify issues with the requirements and design, we drive change in the purpose and implementation of the software. If the tests aren’t adapted to match the evolution of the software, how useful will the tests be? Continuing to follow a plan once it has been established that the plan no longer reflects the situation on the ground is an exercise in futility. The tests must evolve with the software and with our understanding of it. And does every such change require another approval? This creates change inertia, which slows down testing, lengthens the feedback loop and drags down the value of testing.

Where does this desire originate?  In many cases, though demonstrating a lack of understanding as to the nature of discovery, it can be perfectly innocent. However, it can also be a warning signal for something far more sinister:

  • Perhaps the stakeholders of testing have little confidence in the testers: “We want to approve your tests because we don’t believe you are capable of making the right decisions as to how or what to test”. Now, it may be that in such situations the stakeholders have a crystal ball and can predict which tests are required to find all the important bugs, but it is rather more likely that this is evidence of perceived commoditization and the death of trust. If you don’t trust your testers, then replace them – or do a better job of articulating what’s important to you.
  • Perhaps the testers themselves are seeking approval as a means to protect themselves from a witch hunt: “We fear that you will haul us over the coals if we miss a bug – so we want you to approve our tests”. Developers can make bugs but woe betide the tester who misses one? It’s no secret that it impossible to test everything: missing bugs is inevitable. If this is typical of your work environment, take it as indicative of the state of relationships in your workplace. Put some serious thought into whether this can be fixed, or whether you should be considering other options.

Finally it’s worth mentioning that in some cases (for example outsourced testing), both scripts and approvals are a contractual requirement. In these cases it’s worth running through the contract to determine exactly what’s required, and giving thought to how to prevent this from dumbing down the testing.  And have a conversation with however writes the contracts!

More to follow…

Devil in the Detail I

Posted on by
2

Previously…

As a test service provider, the most frequent arguments that I hear in favour of detailed scripts concern review and approval. I’ll start with those.

Argument 1: “You need scripts so that we can review what you intend to test”.

This argument expresses two things: a desire and an assumption:

  • DESIRE: We want to understand what you plan to test.
  • ASSUMPTION: We can only do that if there are scripts.

As far as the desire goes, I love it when my stakeholders take an interest in testing. This is a GOOD thing. I want their input and I’m all for review. The more stakeholders that I can get involved in discussing testing the better:  I want to know how they intend to use the software and what keeps them up at night.

Often, this desire stems from a need to simply understand what the testers will be doing, or from a need to gain confidence that the testers are looking at the right things. Yet sometimes there can be a dangerous side to this desire. It is human nature to crave certainty in an uncertain world, and this can manifest as unrealistic expectations as to what can be planned in advance. I’m sure you’ve seen this: huge and elaborate planning efforts that deliver thousand line plans that are broken the moment the project starts. The reality of testing is that we discover and uncover as we go. To ignore what you learn and the new tests that this might suggest is to introduce unnecessary constraints on your testing. There’s value in planning, but none in attempting to hold to a plan when it is clear that it does not reflect the facts on the ground. If you encounter this expectation, and your mission involves more than simply performing a set of predefined checks, then you need to reset this expectation.

Now, to the assumption; that scripts are necessary for reviews. I’d argue the opposite; that detailed scripts are the death of a review. If you hit your reviewer with a thousand page dump of step by step test scripts from your test management tool (I call this a QC barf), expect narcolepsy rather than any meaningful input. A simple list of test cases is much more digestible and conducive to engaging stakeholders in a useful conversation. A mind map that captures test ideas (see Christin Wiedemann or Darren McMillan’s blog) might be even better when it comes to finding gaps and thinking up new ideas. Either of these approaches comes with a significantly lower price tag, and far greater flexibility, than detailed scripts.

More to follow…

 

Devil in the Detail: Prologue

Posted on by
7

On my first real testing job I hadn’t a clue what I was doing, and no one around me had any testing experience either. This was when the web was in its infancy (I was still browsing with Lynx and a 2400 baud modem) and it didn’t even really occur to me that there might be books on the subject. 

I did the only sensible thing: I made it up as I went along. I can only assume that I’d subconsciously overheard people talking about testing at some point, because I did know one thing: testing needed scripts. I dutifully set about talking to the team, reviewing the design and writing a set of step by step scripts that covered the requirements. The results were not exactly outstanding, but I muddled through.

Over the next few years, I relied on scripts less: I found that they slowed me down, that I could test more productively without them. Much of the time I’d define lists of tests, or in some cases matrices describing combinations of data. These served me well. I was documenting to a level that someone with knowledge of the software could execute. Whenever I had to hand over test cases, I’d add some supplementary information and sit down and train the new testers on using the software.

On one project, one of my team told me “I prefer exploratory testing to scripts”. I wasn’t really sure what she meant. It wouldn’t be long before I found out: this was shortly after a deeply embarrassing interview had convinced me to start learning more about testing, and I was spending every spare moment reading anything I could find. 

My reaction when I first read about exploratory testing? I couldn’t understand what the fuss was about. This wasn’t anything different or special, this was just testing! Every time I’d ever played with an application to figure out how to test it, I’d been doing ET. Every time I’d prodded and poked at a program to figure out why it wasn’t quite doing what I’d expected, I’d been doing ET. Every time I’d noticed something new and tried out new tests, I’d been doing ET. I found it almost inconceivable that anyone in testing didn’t do this.

As I started managing larger testing teams, I grew to appreciate ET more. I found a surprising degree of script dependence and inflexibility amongst many testers. Encouraging the use of exploratory testing helped to correct that. 

What constantly surprises me is the range of arguments that I came across in favour of detailed “idiot scripts”, scripts that are detailed enough that any idiot could use them. This series of posts will take a look at some of those arguments.

Commoditization: Three Scenarios

Posted on by
6

In the previous post, I introduced the commoditization of testing. This trend devalues the skill of the tester, and has far-reaching consequences for software development as a whole.

How might this play out? What might the future hold for testers?

Here are three possible scenarios.

Scenario 1: Runaway Devaluation

The feedback loop driving commoditization continues at full bore, and having claimed the enterprise the infection spreads to software vendors. Gradually, the public becomes inured to failure, with many vendors seeing low software quality as a viable business model.

Testing failures occur regularly, but few executives connect this with diminishing skills. Testing is simply another problem to offload to third parties, and if that fails, to a different testing service provider. Meanwhile, the myth that testing is checking continues to ride high. With continued developments in model based testing and test driven development, many conclude that “the testing problem” has finally been solved by developers.

At the same time, growth in user testing and SME testing further erodes territory that formerly belonged to the tester. Eventually, The International Institute of Business Analysts – who already claim “validation” as part of their body of knowledge – subsume any testing not performed by the developer or end user. Testers find themselves caught in a pincer from which there is no escape. In one form or another testing might live on, but testing as a profession is dead.

Scenario 2: Islands of Hope

The commoditization effect continues unabated, largely conquering the enterprise. Whilst this also spreads to software vendors, some see quality as a means to differentiate themselves in a marketplace largely defined by buggy software and long waits in the IVR-hell of technical support.

Silver bullets come and go, each long on promise and short on delivery: none can resolve “the testing problem”. Whilst successful software development is best served by skilled people working together, that answer is too hard for most: many executives reach to standards and process compliance in the hope of a solution. This only accelerates the decline of tester skills.

However, a scattered handful of executives recognize this problem. Perhaps they were testers themselves, have been fortunate enough to work with skilled testers, or simply see things clearly. These individuals cluster, seeking the company of the like minded, gravitating to those organizations that differentiate themselves on quality. In this way a market niche is born, one in which a small number of skilled testers not only survive, but thrive.

Scenario 3: Generational Change

The commoditization of testing continues for years, driven by a generation of IT managers who never witnessed the power of skilled testing.

The ranks of unskilled testers swell. Lacking a demand for skill, few employers are interested in their development. Yet this cannot suppress the human appetite to learn; many testers turn to the Internet in order to develop themselves, and the Internet is owned by the passionate.

Online coaching and training – much of it voluntary – flourishes. Soon, supply outgrows the diminishing demand for skill. Many testers drop out of testing rather than serve as commodity checkers. Some leave the industry altogether, many become developers or project managers. As demographics works its magic, those who believe in a better way find themselves in executive roles. They demand testers with skill; they demand that testing be at the heart of software development. The feedback loop driving commoditization slows and finally reverses as a paradigm shift begins to take hold.

I don’t have a crystal ball, and these scenarios have no real predictive power. However, through the exploration of scenarios such as these, we can gain insights as to how to navigate the future – both as individuals and as a craft. I invite you to add your own.

 

The Commoditization of Testing

Posted on by
10

As testers, one of the greatest challenges we face is the perception that the service we provide is a commodity. This may not be universal, but it is endemic in many parts of the IT industry.

Is software testing a commodity? According to Investopedia; “When a product becomes indistinguishable from others like it and consumers buy on price alone, it becomes a commodity”. In order for testing to qualify as a commodity, testing services must lack qualitative differentiation. Yet testing has many sources of differentiation:

  • Approach. Testing is context dependent. Some testers will bring practices and experiences that are relevant to a project, others will not. Some will attempt to bend project context to their approach, others will adapt to its specific needs. Perhaps one day this will cease to be a source of differentiation, but not today.
  • Location. Testing involves the discovery, trade and transformation of information. When communication channels are open and information is shared, remote –even very remote- testing is feasible. When information is hidden and hallway decisions are the norm, colocation is a huge advantage. Project politics and project culture can have a significant impact on the value provided by testers in different locations.
  • Skills. Testing is a vast and growing discipline, with ample room for specialization. Again, context plays a role: different projects will require testers with different skills. Further, the range of skills required is mind boggling: modeling skills, analysis skills, design skills, communication, relationship and interpersonal skills. Indeed, testing thrives on variety – the range of insights that different people with varied skills, outlooks and experiences bring to bear. The notion that testing is undifferentiated is anathema to testing itself.

In short, not all testing services are equal, nor should they be. Testing is not a commodity.

Then where does this perception come from? It originates from the belief that testing is unskilled. The most obvious act in testing is checking, which in itself requires little skill. In contrast, skilled activities such as modeling, analysis, design and evaluation are harder to observe. This belief is as superficial as believing that the cover is all that makes up a book.

Why is this important? The view that testing is a commodity provided by the unskilled has chilling effects on collaboration, effects that can destroy the mutual interdependence between testers and project.

Let’s consider one possible scenario that illustrates how this can play out. Imagine for a moment that you are a non-testing participant on a project.  Imagine that you see testing as essentially unskilled, a simple matter of translating specifications into scripts and then executing them.

You’re a busy person, and like anyone else on the project you are probably juggling a dozen different things at any given moment. The testers keep calling you, or emailing you, or messaging you, or asking for meetings. Why do they keep bothering you?!? Can’t they just get on with scripting? Can’t they do their jobs?

Testing depends on information. Mere specifications are a remarkably low bandwidth form of communication and on their own are seldom adequate for effective testing. Testers need to ask questions, it’s what they do. Yet by doing so, the tester can fail in the eyes of their stakeholders.

This is getting crazy, the testers won’t give up. Now they are calling the PM, BA and developers. Don’t they know we’ve got deadlines? Don’t they know we’ve got enough of our own work to do, without having to do theirs as well? It’s time to channel this through a handful of people.

A common reaction to the “we’re too busy to answer questions from testers” problem is to enact a communication plan which restricts the flow of information to (and from) testers through a small group of people. Those people may not have direct access to the relevant information, so may in turn have to ask other members of the project, thus continuing the distraction and introducing rounds of questions, answers and clarifications bouncing back and forth. With communications bandwidth restricted, and telephone game distortion, the value of information available to the testers diminished rapidly.

What’s with these testers anyway? Not only are they a nuisance, but now I’ve seen their scripts, they’re missing important stuff! And that bug that got through the other week…this can’t go on. The testers are clearly rubbish, we need to manage them the directly.

At this point, the relationship between the testers and the wider project has completely failed. It is almost inevitable that this will lead to micromanagement, a master/slave relationship that further inhibits the testers’ opportunities for success. The mechanisms for this are varied. Perhaps the absence of trust will lead to demands for excessive documentation. Perhaps process or standards compliance will be enforced, in word rather than in spirit. Perhaps unsound performance metrics will be imposed. Each introduces opportunity costs, degrading the volume and value of the information the testers can provide. Each further distorts tester behavior. Each will increase the real cost of testing. And satisfaction in the testers will spiral ever downwards until the project comes to its naturally end, or the testers are dismissed.

A further implication of commodity testing is the notion that testing services are essentially fungible, i.e. that it does not matter who provides the service, or from where. All that matters is price; this is labor arbitrage. The result is a race to the bottom, with testing service providers in the lowest cost regions of the world the front runner. Remote testing may be successful in some situations, but imagine the scenario above compounded with multiple time zones and cultural factors such as power distance. In such circumstances, success is not an option.

How might things be different? We testers need to be aware of these issues, and to realize that things can be different. We need to continuously develop our skills, to market the importance of them, and to find ways to give visibility to those aspects of testing that are not checking. The alternatives to our craft are unthinkable.

Uncoverage

Posted on by
1

“When we uncover something…we are like investigative reporters, revealing something that would otherwise have remained unknown”. – Understanding by Design, Grant Wiggins and Jay McTighe.

At its heart, testing is about discovery and understanding: we seek to understand our stakeholders and their expectations, we manipulate and observe the behavior (and misbehavior) of the software so that we might discover how it behaves, we adjust and refine our understanding of both as we learn.

I’ve recently been reading Understanding by Design, a book dedicated to putting the understanding back into education. At times, the parallels with testing are striking: of particular note is the distinction that Wiggins and McTighe make between coverage and uncoverage.

The authors describe coverage in education as a negative term “because when content is covered the student is led through unending facts, ideas, and readings with little or no sense of the overarching ideas, issues, and learning goals that might inform study”. They liken it to a whirlwind tour of Europe, with each day being another country with little opportunity to take in much of anything. In contrast, they describe uncoverage as “finding something important”.

Let’s take a moment to look at coverage in the context of testing:

  • Each form of coverage is underpinned by certain techniques, each of which has blind spots. By pursuing coverage with one we forgo opportunities to apply others; we forgo the opportunity to learn what other techniques might teach us.
  • Coverage has a focusing effect. Each technique leads us to observe certain aspects of software behavior at the cost of inadvertently ignoring others, again missing opportunities to discover.
  • Coverage can become a proxy for testing “progress”, resulting in an inexorable march to schedule, further curtailing opportunities to learn. How many times have you found yourself driven to complete a given set of tests at the expense of what you might uncover?

As Wiggins and McTighe point out, coverage confuses means and ends, “our teaching with any resultant learning – mere planting with the yield, or marketing with sales”.

Coverage should never be our goal in testing; it should simply be a means highlighting what remains to be done. Which of the following statements are more useful? “We’ve covered the requirements” OR “There are some risks we haven’t covered yet”. The former tells us essentially nothing about software quality or the effectiveness of our tests whilst the latter gives us something actionable.

When testing, make uncoverage your ends and relegate mere coverage to means.

A Trio of WTFs

Posted on by
6

I’ve recently been taken aback by some of the things that people have said or written about exploratory testing.

First there was this blog post that says of prep time: “And you would see the exploratory test team sitting idle, because there is no software to explore.” The post continues “In a less dishonest comparison, I’m sure the exploratory team would find something useful to do”. Damn right, and I’m glad the author added that: there is a persistent myth that exploratory testing is unplanned testing. Whilst ET can perform admirably with little or no advanced planning, it can benefit from the opportunity to do so. Every time that I’ve mobilized ET on a project where I’ve had test prep time, I’ve used that time to maximum effect: exploring the context, learning from my stakeholders just what they hoped to get out of the software, absorbing every scrap of information I could get hold of about what the software was meant to do, creating models to describe its behavior, figuring out lists of what to test and how– all with the goal of having a rich a set of test ideas for when something executable arrived. Exploratory testing is not unplanned testing, it simply implies that you accept that what you learn will change the plan.

Next, there was rereading Copeland’s A Practitioner’s Guide to Software Test Design. Copeland claims that ET has no power to prevent bugs whereas scripted testing does; the notion of testing the test basis through test design. The former is true of any testing that has the testers showing up on the same day the code does, but –again- given the opportunity of prep time there is absolutely no reason why such prevention cannot be part of an exploratory approach. Asking questions, seeking clarity and building understanding all have a remarkable power to expose ambiguity and contradiction. ET does not preclude this. An exploratory approach can provide prevention.

Finally there was Rex Black’s recent webinar on test strategies (which I found interesting, but that’s a post for another day). Without explicitly referring to exploratory testing, Black positions ET as being one of several “reactive” test strategies. Now, the term reactive could be taken to have negative connotations, but let’s disregard the realm of management speak that defines the opposite of reactive as proactive and think in terms of plain English. You know? The opposites of reactive include unreactive and unresponsive. I can live with that: ET is reactive in exactly the same way that a corpse isn’t. Anyone else getting an image of moldy dead test scripts? Then there’s the idea that I will know when I am employing a “reactive strategy” because I will be applying my experience rather than a testing technique such as domain testing etc. When did techniques and experience become mutually exclusive? It seems that whenever I’ve applied formal testing techniques during a test session that I wasn’t doing ET at all…silly me. An exploratory approach may be reactive (and that is a good thing), but it does not preclude any test technique.

Selecting Strategies

Posted on by
Reply

Previously on Exploring Uncertainty…

In the previous post in this series I discussed situational analysis: one half of the test strategy equation. In this post I’m going to discuss the other side of the coin: strategic planning.

Again, there’s no formula for this. I’m not going to attempt to outline any creative process for generating strategy ideas, nor am I going to describe methods for choosing between them, because, well, you can Google just as well as I can.

Instead, this post (much like the last) will describe a mixed bag of things that I find useful.

Differentiate between long range and short term strategy decisions. Some strategies are long term investments, for example certain forms of automation. Other aspects of strategy will be in constant flux, for example test design techniques that you select during a test session. Do not confuse the two: committing ALL your strategic decisions in a monolithic document and then refusing to change is a recipe for disaster. If the plan stinks, then trying harder to deliver against that plan is insanity: CHANGE THE PLAN. Context is not static: projects unfold over time in ways that are often not predictable. Recognize this, and the need to course correct as your landscape changes.

Establish mission, goals and constraints. Test strategy ideas should be viewed through the lens of your mission. Why are you testing and what kind of information are you seeking? Will your strategy support that? What if there are multiple goals? Could they ever be in conflict? Which are more important than others? How do your strategy ideas stack up against each goal? What constraints are you operating under? Are your ideas unrealistic in light of your constraints? Are the constraints fixed, or are they negotiable?

Consider plenty of alternatives. Don’t fixate on the first idea that springs to mind. Generate lots of ideas. Get other people involved: they’re allowed to have ideas too, and some will be better than yours. Bounce ideas off one another and think of variations on each one. Consider under what contexts each strategy might be a good idea, and under what contexts it might not; then consider fit with the current context.

Test your ideas. Avoid making a psychological commitment to your ideas, and open yourself up to challenge. Each challenge will either invalidate or strengthen your ideas. You are a tester; you test things. That can include more than just software; your ideas can be tested too:

  • Prototype. Remember that strategy is not a linear process of analysis followed by design. Get your thoughts on strategy out there to be tested by your stakeholders. Manage their expectations so they know these are your “initial thoughts” and then solicit feedback. Socialize new ideas informally before trying to reach formal agreements. This process can reveal new things about your context.
  • Use simulations. Imagine a range of bugs; would your strategy catch them? What weaknesses does this expose?
  • Play “What if?” What if your assumptions fail? What if a critical risk is realized? What if a key stakeholder leaves? What if the goals change? What if the constraints change? Explore how sensitive your ideas are to changes in context. Don’t forget the role of luck. What’s the worst thing that can happen? Would your strategy fail? What’s the best thing that can happen? You’re not relying on that are you?
  • Consider scenarios. A variation of “What If?” Read “The Art of the Long View” and learn how to use scenario planning – use what you know about your context to create scenarios as to how the project might pan out. Ask yourself how your ideas perform under each scenario.

Accept that sometimes you are under time pressure. The above is all well and good, but sometimes you will need to act quickly: you can’t take five days to decide on a strategy for a test session that starts in five minutes. Recognize that perfection can be an obstacle to attaining good enough, and only invest the time that you can afford in decision making. Ask yourself what the consequences are of not making a decision by a given deadline, versus the consequences of making a bad decision. Determine what information you have available to you right now; versus new information you might have later. Can your decisions be deferred until you have more information, or do you need to act now?

Have fun. Test strategy is like juggling fifteen pineapples, a chainsaw and a porcupine: it’s a challenge, there’s a lot going on, and there’s always the possibility of a mess. But because of that challenge it can be thoroughly rewarding. Enjoy.

Not Checking

Posted on by
7

Michael Bolton’s distinction between testing and checking seems to have had some unintended consequences. Don’t get me wrong: I have found immense value in this distinction. It’s just that there is some confusion that needs clearing up.

The first confusion relates to value. Some of us (myself included) have fallen into the trap of undervaluing checking. I wrote about this in Checking is Not Evil, and I’m not going to explore that any further today.

The second confusion relates to the relationship between testing and checking. I’ve seen some of this locally, and it is also apparent in some of the comments to Checking is Not Evil. It’s easy to read blog post titles like “Testing vs. Checking” (and possibly reading no further) whilst imaging testing and checking as a Venn diagram that looks like this:

 

 

 

 

 

That would be a mistake. As Michael points out in that self-same post, testing involves checking. In other words: checking is something that we can do as part of testing. It is a subset:

 

 

 

 

 

The third confusion relates to the content of the-bit-of-testing-that-is-not-checking. One response I’ve run into involves some vague hand waving and allusions to mystical powers that I know about but am not going to share with you. Another is something along the lines of “well obviously that’d be exploratory testing”. There is some truth in this, in that many of the things that make for good ET will fall under this category, but more generally this answer is not a good fit: ET is an approach, a style of testing. It’s definition is too narrow to encompass all of testing less checking.

So what IS the-bit-of-testing-that-is-not-checking? It is what you – as a human being – bring to testing: any skill, any mindset, and any insight that serves your testing mission. You might have skills at identifying checks or automating them, you might have a nose for trouble or a mind made for modeling, you might be a master relationship builder. That-which-is-testing-but-is-not-checking is anything and everything that you can do that a tool cannot, anything and everything that you can bend to the purpose of investigating software so as to provide information about it.