A Reflection

If your testing doesn’t provide value, this may be what you get… 

Testing is the culprit of project delays and cost overruns. Testers waste time, cost a fortune and add no value. And it seems that they are getting uppity these days, talking about testing like it’s interesting, reading blogs and Tweating. It’s time to put them back in their place.

Firstly, why can’t they just settle on a process? In my day it was simple: plan it, script it, test it. Why can’t testers fit in these days? I think I’ll make them write up a process that tells them exactly what to do on every project. Or perhaps I’ll get some consultants in to do the job, they’ll know what’s best. Once we have the process, I’ll bring the auditors in – they can give the process their blessing and make sure the testers are complying to the letter. Testers watch out. That should put the fear of God into you, make you toe the line. What’s that you say? Context? Bollox. Cookie cutter works for McDonald’s, and they make a fortune.

And while we’re at it, why stop at process? If you think about it, every test is a bloody process! The testers can damned well create detailed scripts for each and every test. That way we can make sure they’re testing the right stuff and even better it’ll make them “fungible”. Don’t tell them I said this…but once we’ve got scripts then anyone can test! Imagine that, we could juniorize like crazy and bring in any idiot off the street. Adaptability? Learning? IMAGINATION? What crap are you talking? There’s no room for imagination in testing, testers can learn all they need from the specifications, and they wouldn’t need to be adaptable if they could only plan worth a damn.

While I’m on the subject, we need to beef up test planning…remind me to include that when we get the consultants in. I mean, how hard can it be to create a template? Hell, give me five minutes and I’ll Google one myself. And we’ll need spreadsheets, LOTS of spreadsheets and some kind of estimating formula. Unrealistic huh? Nah, it’s no big deal: we’ll just start counting how many scripts the testers can execute per day.

Now THAT’s a thought! If we measure that we can sort the wheat from the chaff and ditch all the non-productive testers! OK, you’re right, I agree that execution rate isn’t the only important thing. We’ll have to measure a bunch of other things as well, like how many defects they miss and how many times they make mistakes. We can even incentivise it with a tester of the month certificate, a pat on the back and a “well done, you’re not fired…this time!”

Am I on a roll today or what? What other brilliant ideas can I come up with? Got it! The thing that bugs me the most is all the questions, the constant “me, me” distractions from the test team. Everyone knows testers should be independent, so the more independent the better right? Let’s put them on a different floor and tell them that they’re not allowed to talk to the developers! Hell, why stop there?!? Let’s offshore the whole lot of them, send them to the otherside of the world! Let’s face it, testing is useless anyway: it’s just a placebo to satisfy the auditors and execs. Well, if I’m going to buy sugar pills, I may as well get ’em cheap.

What? You quit? Whatever, guess I’ll just have to make the devs automate the problem away.


I’ve been noodling with this post for a while now, never quite finishing it off, and then yesterday Dorothy Graham posted Is it dangerous to measure ROI for test automation?, spurring my reply: Yes Dot, yes it is.

All too often I’ve seen ROI used in attempt to either sell or justify test automation. The reasoning goes like this: replace a bunch of manual tests with automated ones, it’ll save money. This is an argument based on a great big stinking ROLie. Automated checks are never a replacement for tests conducted by people. Tests and automated checks are not substitute goods. Machines cannot observe anything other than that which they’ve been programmed to check, their judgment is limited to the algorithms with which they have been provided, they are incapable of insight or intuitive leaps, and they can never play a hunch.

Even when used honestly, as a sanity check of investment decisions, this kind of thinking is perverse. As Dot states: “If we justify automation ONLY in terms of reduced human effort, we run the risk of implying that the tools can replace the people.” In other words, we can perpetuate the myth that testing is a commodity made up of low skilled and easily replaceable parts.

I do not believe that R.O.I. is entirely irredeemable. Such cost comparisons can make sense, but only if applied below and across tests, at the level of testing tasks. For example, earlier this year, I needed to make some tooling decisions relating to the testing of a large volume and highly complex ETL process. Rather than evaluating which tests could be replaced with automation, I instead looked at which tasks should be performed by people, and which should be performed by a machine. Here’s a thumbnail sketch of the reasoning:

  • Data mining, repetitive application of rules: automate.
  • Data conditioning: conceptually possible to automate but insanely expensive to do so exhaustively: let’s stick with people.
  • Expected result generation, high volume repetitive application of rules: automate.
  • Result reconciliation, literally billions of checks per data load: automate.
  • Bug isolation and reporting, investigation and judgment required: need people.

Of course, other things factored into the decision making process (I’ll discuss that a little more at CAST 2012 and in a subsequent post), but having realized that I was beginning to lean heavily towards using large scale mechanized checking to assist with much of our testing, I wanted to carefully check my thinking. In this case, rather than seeking justification, I needed wanted to answer one simple question: Am I making idiotic use of my customer’s money? ROI served as a test of this aspect of my strategy.

Now, this is a narrow example. The parallel nature of ETL test execution lends itself to breaking out and batching the tasks that make up individual tests. For many types of testing this is impossible, and ROI useless. We need a different paradigm, a focus on value instead of replacement. This is fairly straightforward. There are many ways in which tooling can add value: it can enable testers to obtain information that would be inconceivable without tools, it can improve the accuracy and precision of information that they can access and it can enable them to provide information faster or more cost effectively. The tricky part is putting a price on that value so as to determine if a particular automation effort is a worthwhile investment. So why not simply ask? Why not simply discuss the likely value with your customer, and ask what price they would put on it? For example:

  • “This part of the application has a high number of permutations and we can only scratch the surface without automation. What price would you put on being able to identify some major problems in there? What price would you put on knowing that despite our best efforts we haven’t found any major problems with it?”
  • “The current regression checks take about a week to complete. What price would you put on being able to complete them within a few hours of a new build?”
  • “Using flags and stopwatches, we can only scale the performance tests to around 100 testers. What price would you put on a more realistic simulation?”

This might lack the appearance of objectivity that accompanies ROI, but let’s face it; the typical ROI is so speculative and riddled with estimating error as to be laughable. What this approach provides is a quick and easy way of getting to the business of value, and focusing on what tooling can do for our customers rather than only what it will cost them.

Looking Inward, Looking Outward

Testers are not unfamiliar with quality: it is central to the information that we seek to discover and share. However, whilst we spend much time considering those characteristics that make up software quality, perhaps we are a little less attentive to the quality of our own testing. Sure, we regularly discuss principles and practices, and whilst these might hold some interest to the tester, these are often seen as mundane matters – especially to our customers. Customers who are often crying out for information, who want their software tested, but have little interest in how it is done. Do you care how electricity is generated and transmitted? About how a power station is designed or how many miles of cable are needed in the distribution network? Or do you simply care that the lights turn on when you flick the switch?

Now, we testers may not “assure” the quality of software, but we had better damn well assure the quality of our own work. Perhaps we should consider what constitutes “quality testing”. So what is “quality testing”? To answer this, let’s turn first to Weinberg: Quality is value to some person. The quality of a product or service is determined by the value it brings to its customers. We need to look to our customers and understand what they value.

I’ve recently been working with some clients to create testing teams, and a desire to understand value has been central to this. This led me to reflect on the value of what we do and how we do it. Or, to put it another way, what characteristics make up the quality of testing?

First, let’s look at the “what” of testing. What we provide is information, that is our raison d’être. But in what ways does that information provide value? Here are some thoughts:

  • Relevance. Is the information we provide relevant to the needs of our customers? Does it help them answer the questions that they need to answer?
  • Accuracy. Is the information we provide accurate and at an appropriate degree of precision?
  • Contemporaneity. Do we provide information at the right time, when our customers need it? Do we provide feedback quickly, contemporaneous with the moment of creation?

Sometimes it is not only the information that we provide that adds value. Our customers sometimes place a value on how we produce information and often the “how” influences or constrains that information. Let’s look at the “how”:

  • Transparency. Do we provide transparency, visibility into our work? Do we welcome input by our customers so as to afford them the opportunity to fine tune the value that they will gain?
  • Adaptability. Information needs are not static: they evolve throughout the lifecycle of a project, and more generally throughout the lifecycle of a software product. Are we sufficiently adaptable? Do we have the flexibility to change course as we learn more about the software, or as the needs of our customers change? Are we able to take this in our stride or will we fall victim of mission creep? Can we stay relevant?
  • Sustainability. As we evolve our service, deepening our understanding of the software and the needs of our customers, are we capable of preserving and sustaining that knowledge? Can we ride out the inevitable changes in team members?
  • Compliance. Sometimes our customers have regulatory, audit or standards driven requirements. Can we fulfill the obligations these bring? Can we do so without compromising the value we might otherwise provide?

Of course, our ability to provide value on any of the above dimensions is constrained by time and cost. Tradeoffs have to made, and it is your customers who need to decide which ones. To quote Weinberg again: If you don’t care about quality, you can achieve any other requirement. Do your customers care about the quality of the testing you perform? If so, perhaps a first step to making appropriate tradeoffs is to explore with your customers what quality means to them. I invite you to look inward at the quality of your testing, and to look outward at what your customers value. Let me know what you find.

Devil in the Detail III


A continued discussion about the arguments for detailed test scripts…

Argument 3: “We need scripts so that anyone can test”.

Did you know that anyone can win a medal in the Olympic 100m? Here’s how:

  1. Wait for starting pistol to fire.
  2. Put one foot in front of the other, and repeat really fast.
  3. Collect medal.

Easy! Well of course it isn’t that easy. I suspect that were I to follow that script I’d fall on my face, pull a muscle, or – just maybe – come in last, wheezing.

So how might a tester fall on their face when following a script? What if the software does what the script says, is that a “pass”? Software can misbehave in a near infinite number of ways. It is impossible to encode checks for every instance of system behavior that might be considered a problem. Even were that possible, our stakeholders would balk at the costs of developing, executing and maintaining such checks. Yet a tester following a script is focused on those aspects of behavior that are encoded as checks. They are not primed to observe anything else, nor are they given the means to evaluate any other behaviors. Should anything else occur, they are likely to miss it, else not know what to do with it.

What if the software doesn’t do what the script says, is that a problem? It might be. It might suggest that the person who encoded the check might consider it a problem. Or that it is a legitimate aspect of system behavior not accounted for in the setup of the check. Or that some other condition not encoded in the script has been satisfied. Or that the tester has interpreted the script in a manner other than which it was intended. Again, the tester who relies on a script does not have the means to evaluate any of this. For some further thoughts on this topic, see Doug Hoffman thoughts as to why tests don’t really pass or fail.

There is a significant gap between following a script and the ability to perform such that the desired results of a script are achieved. In the example of the 100m sprint that gap is obvious: it captures only the race itself whilst completely ignoring training and diet. Like sprinters, testers need to train. We need to train our observation skills, our thinking skills, our communication and relationship skills, our information acquisition and synthesis skills. The skills that can help us are many and varied. Also like sprinters, we need the right diet, but I’m not talking about pizza and pop delivered to the lab at silly o’clock at night. The diet testers need is a steady diet of information, and as in the world of nutrition we need to digest the right stuff. Now, some people believe that scripts provide a means for testers to learn how to test a particular solution. Quite apart from scripts being a lousy way to learn, this belief demonstrates a seriously blinkered view as to the kinds of information a tester needs. Scripts might tell you what buttons to press and in which order, they might give you some clues (with the aid of some inference) as to what results to expect when you press those buttons, but they give you no indication of why those results might be important, or what the consequences are of a different result. Nor do scripts provide any insight into any of the human aspects of the project. Scripts are the equivalent of junk food.

So why is it important that “anyone can test”? Sometimes this reflects a desire to mitigate the risks of critical people leaving the project. This is an entirely reasonable concern for which scripts – though an easy answer – do nothing to address. In other cases, this is born of the desire to reduce the costs of testing: if we can get scripts written perhaps we can “juniorize” the team or outsource the testing. This is a case of commoditization. Recently it was suggested to me that projects should always seek to use the lowest cost testers, because testing is “just an overhead anyway”. If you really believe that, please put your money where your mouth is and don’t test your software. While you are at it, I suggest that you update your resume.

The Heart of Testing

There are many definitions of testing, some excellent, some banal. Most hinge on its purpose, its deliverable or its activities, yet such descriptions feel inadequate somehow, as if providing only a fleeting glimpse into our craft. They fail to tell the story of what is going on at the heart of testing.

Testing, I believe, is about learning.

To begin with, there is learning about the software. What does it do? How does it behave in a myriad of different situations? As Pete Walen recently remarked: “Testing is neither confirmatory nor destructive.  Instead, testing is an interrogative activity that allows us to discover the true behavior of the software”. I couldn’t agree more, though this is perhaps the most obvious of the areas in which we must learn. Discovery is only part of testing. Try this out: stop reporting bugs, start reporting the results of your every observation as to how software reacts to varying stimuli, enumerate each case in painstaking detail. Any value that you might bring will be lost in the noise.

Testing is not merely a technical investigation, it requires judgment concerning the ways in which software adds to or detracts from value. And how do we do that? Value is relative, it is subjective. To understand the value of software we must first learn about the values of our stakeholders. What does “quality” look like to them? What things are important, what do they hold dear? What might be an unacceptable compromise, to what are they indifferent?  Together, our discoveries about the software and about our stakeholders enable us to draw inferences, to make judgments, as to the value that software provides. Together, perhaps, these two aspects of learning provide a useful portrayal of testing, yet to be a successful tester there are many more aspects of learning. We must be capable of learning at many different levels besides.

How do we learn about software? How do we learn where to look, or more accurately where to look to find the problems that will matter most? How do we actually learn about the values of our stakeholders? How do we learn the levers to pull in order to establish credibility or legitimacy? To learn these things, we much learn something else: we must learn how to interact with our stakeholders, regardless of how technical they are, regardless of how articulate they are, regardless of how time pressured they are, regardless of whether they give a damn about the finer points of testing. Every project has its own culture, its unwritten rules, its norms and power relationships. Every project will be different: there is no process, methodology or communications plan that will tell you how to do this. You need to immerse yourself in the project and learn how to interact, much as an immigrant needs to learn how to interact with natives of their host culture.

And if this were not enough, the range of skills and knowledge that a tester should, or could, apply is vast beyond comprehension. We constantly find ourselves learning new skills, even inventing or reinventing them. No amount of learning can be considered enough, so we must become experts at learning, and this requires that we learn about learning itself. How do we quickly acquire and assimilate new information and skills? How do we apply them? How do we choose what to learn next?

Learning lies at the heart of testing. Viva the learning tester.