Tag Archives: Opinion

Dear Paul


I’d like to thank you for your kind words with regards my recent post. I agree with your assertion that there are a number of factors at work that will influence whether a tester will notice more than a machine, and I’d love to know more about your case study.

I suspect that we are closely aligned when it comes to machine checking. One of the main benefits of making the checking/testing distinction is that it serves to highlight what is lost when one emphasizes checking at the expense of testing, or when one substitutes mechanized checks for human ones. I happened to glance at my dog-eared copy of the Test Heuristics Cheat Sheet today, and one item leapt out at me: “The narrower the view, the wider the ignorance”. Human checks have a narrower view than testing, and mechanized checks are narrower still. We need to acknowledge these tradeoffs, and manage them accordingly.

I think we need to be careful about the meanings that we give to the word “check”.  You say the usage that you have observed the most is when “talking about unmotivated, disinterested manual testers with little domain knowledge” or when “talking about machine checking”. Checking, in and of itself, is not a bad thing: rather checks are essential tools. Further, checking, or more accurately the testing activities that necessarily surround checking, are neither unskilled nor unintelligent. Not all checks are created equally: the invention, implementation and interpretation of some checks can require great skill. It is, in my opinion, an error to conflate checking and the work of “unmotivated, disinterested manual testers with little domain knowledge”. Testers who are making heavy use of checking are not necessarily neglecting their testing.

More generally, I worry about the tendency – conscious or otherwise – to use terms such as “checking testers” (not your words) as a pejorative and to connect checking with bad testing. I would agree that the inflexible, unthinking use of checks is bad. And I agree that many instances of bad testing are check-heavy and thought-light. But rather than labeling those who act in this way as “bad testers”, and stopping at the label, perhaps we should go deeper in our analysis. We do after all belong to a community that prides itself in doing just that. I like that, in your post, you do so by exploring some traits that might influence the degree to which testers will go beyond checking.

There are a multitude of reasons why testers might stop short of testing, and it seems to me that many of them are systemic. Here’s a few to consider, inspired in part by Ben Kelly’s series The Testing Dead (though far less stylish). It is neither exhaustive nor mutually exclusive:

  • The bad. Some people may actually be bad testers. It happens; I’ve met a few.
  • The uninformed. The testers who don’t know any better than to iterate through checks. It’s how they were raised as testers. Checking is all they’ve been taught; how to design checks, how to monitor the progress of checks, how to manage any mismatches that the checks might identify.
  • The oppressed. The testers who are incentivized solely on the progress of checks, or who are punished if they fail to hit their daily checking quotas. Testing is trivial after all: any idiot can do it. If you can’t hit your test case target you must be lazy.
  • The disenfranchised. Ah, the independent test group! The somebody-else’s problem group! Lock them in the lab, or better yet in a lab thousands of miles away, where they can’t bother the developers. If fed on a diet of low-bandwidth artifacts and divorced from the life, the culture of the project, is it any wonder then that their testing emphasizes the explicit and that their capacity to connect observations to value is compromised?
  • The demotivated. The testers who don’t care about their work. Perhaps they are simply uninformed nine-to-fivers, perhaps not. Perhaps they know that things can be different, cared once, but have given up: that’s one way to deaden the pain of hope unrealized. Many of the oppressed and disenfranchised might find themselves in this group one day.

Do you notice something? In many cases we can help! Perhaps we can encourage the bad to seek alternate careers. Perhaps we can help the uninformed by showing them a different way (and as you are an RST instructor, I know you are doing just that!). Perhaps we can even free the oppressed and the disenfranchised by influencing the customers of testing, the decision makers who insist on practices that run counter to their own best interests. That might take care of some of the demotivated too.

I like to think there is hope. Don’t you?

Kind regards,



On April 6th 2009, an earthquake devastated L’Aquila, a town in central Italy. It killed more than three hundred people, injured thousands and displaced tens of thousands. This tragedy continues to have repercussions: last month, a group consisting of six scientists and a government official were convicted of manslaughter on the grounds that they failed to properly represent the risks of such an event.

These convictions have drawn considerable media attention, much of it contradictory, some of it misleading. One might be forgiven for believing the absurd: that these men have been found guilty of failing to predict an earthquake. The reality seems to be more complex. However, my intention with this post is simply to draw attention to lessons that this case may hold for the tester. This is neither a detailed account of the facts of the case, nor is it an opinion piece concerning the rights and wrongs of the Italian legal system. If you are seeking such things, then please Google your heart away: you’ll find plenty of accounts and opinion online.

The first thing to strike me about this case is the different opinions as to the types of information scientists should be providing about earthquake risks. In the week prior to the earthquake, these scientists met to assess the probability of an earthquake taking place, given an unusually high level of seismic activity in the preceding weeks. The scientists may have believed that providing such an assessment was the extent of their role: according to an official press release from Italy’s Department of Civil Protection, the purpose of this meeting was to provide local citizens “with all the information available to the scientific community about the seismic activity of recent weeks”. However, a key argument in the prosecution’s case was that the scientist had a legal obligation to provide a risk assessment that took into consideration factors such as population density, the age and structure of local housing etc. There is a world of difference between assessing the probability of an event and conducting risk assessment.

Does this sound familiar? Have YOU ever run into a situation where testers and their customers have conflicting views as to the role of the tester? I see these things playing out pretty much every day. Testers providing “sign off” vs. “assuring quality” vs. “providing information”: these are well known and well publicized debates that are not going to go away any time soon. Ignoring these issues and allowing such expectation gaps to persist is to court disaster: they erode and destroy relationships and it is the tester who will lose when unable to live up to the expectations of those they serve. Nor is toeing the line and trying to keep everyone happy a solution. Often testers must deliver difficult messages, and it is impossible to do so whilst playing popularity games. Imagine being the tester who has been cowed into “signing off” on a product that ends up killing someone or causing a financial loss. If you do this, then you deserve what is coming to you.

Now, whilst I strongly subscribe to the view that our role is as information providers, I have noticed something disturbing lately: testers who seem to feel that their responsibility ends with adding smiling/frowning faces to a dashboard or filing bug reports, “we reported the bug so now it’s not our problem”. This is wholly inadequate. If our role is to provide information, then handing over a report is not enough. Providing information implies not only acquiring information, but communicating it effectively. A test plan is not communication. A bug report is not communication. Even a conversation is not communication. Only when the information borne by such artifacts is absorbed and understood has communication taken place. This is neurons not paperwork. I recently had a conversation with a tester who objected to articulating the need to address automation related technical debt in a way that would be understood by project executives. Perhaps he thought this was simply self-evident. Perhaps the requests of testers should simply be accepted? Perhaps the language of testers is easily understood by executives? I disagree on all counts: testers need to be master communicators, we need to learn how to adapt our information to different mediums, but most importantly we need to learn to tailor our message to our many different audiences. Facts rarely speak for themselves; we need to give them a voice.

Another aspect of this case that I find interesting is that it seems that Bernardo De Bernardinis, the government official who has been convicted, may have had a different agenda from the scientists: to reassure the public. He had motivation to do so: not only had a local resident been making widespread, unofficial, earthquake predictions, but in 1985 Giuseppe Zamberletti, a previous head of the Department of Civil Protection, was investigated for causing panic after ordering several, and in hindsight unnecessary, evacuations in Tuscany. Before the above meeting took place he told journalists that everything was “normal” and that there was “no danger”. This advice had fatal results: many of the local inhabitants abandoned their traditional earthquake precautions in the belief that they were safe.

This is the kind of reassurance that science cannot, that scientists should not, give. It is the same with testing. Have you ever worked for a project manager, product owner or customer who simply wanted to know that everything would be okay? Of course you have, this is human nature: we crave certainty. Unfortunately, certainty is not a service we can provide. Not if we want to avoid misleading our customers. Not if we value integrity. We are not in the confidence business: we are no more Quality Reassurance than we are Quality Assurance.

Science and testing are often misunderstood, and the customers of both have needs and expectations that cannot be fulfilled by either. Scientists need to do a better job of communicating the limits of what they can do. Testers need do the same. In the L’Aquila case, the prosecution stated that the accused provided “incomplete, imprecise, and contradictory information”. Often information IS incomplete, imprecise and contradictory. Scientists and testers alike would be well advised not to hide the fact, but to frequently draw attention to it.

A Reflection

If your testing doesn’t provide value, this may be what you get… 

Testing is the culprit of project delays and cost overruns. Testers waste time, cost a fortune and add no value. And it seems that they are getting uppity these days, talking about testing like it’s interesting, reading blogs and Tweating. It’s time to put them back in their place.

Firstly, why can’t they just settle on a process? In my day it was simple: plan it, script it, test it. Why can’t testers fit in these days? I think I’ll make them write up a process that tells them exactly what to do on every project. Or perhaps I’ll get some consultants in to do the job, they’ll know what’s best. Once we have the process, I’ll bring the auditors in – they can give the process their blessing and make sure the testers are complying to the letter. Testers watch out. That should put the fear of God into you, make you toe the line. What’s that you say? Context? Bollox. Cookie cutter works for McDonald’s, and they make a fortune.

And while we’re at it, why stop at process? If you think about it, every test is a bloody process! The testers can damned well create detailed scripts for each and every test. That way we can make sure they’re testing the right stuff and even better it’ll make them “fungible”. Don’t tell them I said this…but once we’ve got scripts then anyone can test! Imagine that, we could juniorize like crazy and bring in any idiot off the street. Adaptability? Learning? IMAGINATION? What crap are you talking? There’s no room for imagination in testing, testers can learn all they need from the specifications, and they wouldn’t need to be adaptable if they could only plan worth a damn.

While I’m on the subject, we need to beef up test planning…remind me to include that when we get the consultants in. I mean, how hard can it be to create a template? Hell, give me five minutes and I’ll Google one myself. And we’ll need spreadsheets, LOTS of spreadsheets and some kind of estimating formula. Unrealistic huh? Nah, it’s no big deal: we’ll just start counting how many scripts the testers can execute per day.

Now THAT’s a thought! If we measure that we can sort the wheat from the chaff and ditch all the non-productive testers! OK, you’re right, I agree that execution rate isn’t the only important thing. We’ll have to measure a bunch of other things as well, like how many defects they miss and how many times they make mistakes. We can even incentivise it with a tester of the month certificate, a pat on the back and a “well done, you’re not fired…this time!”

Am I on a roll today or what? What other brilliant ideas can I come up with? Got it! The thing that bugs me the most is all the questions, the constant “me, me” distractions from the test team. Everyone knows testers should be independent, so the more independent the better right? Let’s put them on a different floor and tell them that they’re not allowed to talk to the developers! Hell, why stop there?!? Let’s offshore the whole lot of them, send them to the otherside of the world! Let’s face it, testing is useless anyway: it’s just a placebo to satisfy the auditors and execs. Well, if I’m going to buy sugar pills, I may as well get ’em cheap.

What? You quit? Whatever, guess I’ll just have to make the devs automate the problem away.


I’ve been noodling with this post for a while now, never quite finishing it off, and then yesterday Dorothy Graham posted Is it dangerous to measure ROI for test automation?, spurring my reply: Yes Dot, yes it is.

All too often I’ve seen ROI used in attempt to either sell or justify test automation. The reasoning goes like this: replace a bunch of manual tests with automated ones, it’ll save money. This is an argument based on a great big stinking ROLie. Automated checks are never a replacement for tests conducted by people. Tests and automated checks are not substitute goods. Machines cannot observe anything other than that which they’ve been programmed to check, their judgment is limited to the algorithms with which they have been provided, they are incapable of insight or intuitive leaps, and they can never play a hunch.

Even when used honestly, as a sanity check of investment decisions, this kind of thinking is perverse. As Dot states: “If we justify automation ONLY in terms of reduced human effort, we run the risk of implying that the tools can replace the people.” In other words, we can perpetuate the myth that testing is a commodity made up of low skilled and easily replaceable parts.

I do not believe that R.O.I. is entirely irredeemable. Such cost comparisons can make sense, but only if applied below and across tests, at the level of testing tasks. For example, earlier this year, I needed to make some tooling decisions relating to the testing of a large volume and highly complex ETL process. Rather than evaluating which tests could be replaced with automation, I instead looked at which tasks should be performed by people, and which should be performed by a machine. Here’s a thumbnail sketch of the reasoning:

  • Data mining, repetitive application of rules: automate.
  • Data conditioning: conceptually possible to automate but insanely expensive to do so exhaustively: let’s stick with people.
  • Expected result generation, high volume repetitive application of rules: automate.
  • Result reconciliation, literally billions of checks per data load: automate.
  • Bug isolation and reporting, investigation and judgment required: need people.

Of course, other things factored into the decision making process (I’ll discuss that a little more at CAST 2012 and in a subsequent post), but having realized that I was beginning to lean heavily towards using large scale mechanized checking to assist with much of our testing, I wanted to carefully check my thinking. In this case, rather than seeking justification, I needed wanted to answer one simple question: Am I making idiotic use of my customer’s money? ROI served as a test of this aspect of my strategy.

Now, this is a narrow example. The parallel nature of ETL test execution lends itself to breaking out and batching the tasks that make up individual tests. For many types of testing this is impossible, and ROI useless. We need a different paradigm, a focus on value instead of replacement. This is fairly straightforward. There are many ways in which tooling can add value: it can enable testers to obtain information that would be inconceivable without tools, it can improve the accuracy and precision of information that they can access and it can enable them to provide information faster or more cost effectively. The tricky part is putting a price on that value so as to determine if a particular automation effort is a worthwhile investment. So why not simply ask? Why not simply discuss the likely value with your customer, and ask what price they would put on it? For example:

  • “This part of the application has a high number of permutations and we can only scratch the surface without automation. What price would you put on being able to identify some major problems in there? What price would you put on knowing that despite our best efforts we haven’t found any major problems with it?”
  • “The current regression checks take about a week to complete. What price would you put on being able to complete them within a few hours of a new build?”
  • “Using flags and stopwatches, we can only scale the performance tests to around 100 testers. What price would you put on a more realistic simulation?”

This might lack the appearance of objectivity that accompanies ROI, but let’s face it; the typical ROI is so speculative and riddled with estimating error as to be laughable. What this approach provides is a quick and easy way of getting to the business of value, and focusing on what tooling can do for our customers rather than only what it will cost them.

Commoditization: Three Scenarios

In the previous post, I introduced the commoditization of testing. This trend devalues the skill of the tester, and has far-reaching consequences for software development as a whole.

How might this play out? What might the future hold for testers?

Here are three possible scenarios.

Scenario 1: Runaway Devaluation

The feedback loop driving commoditization continues at full bore, and having claimed the enterprise the infection spreads to software vendors. Gradually, the public becomes inured to failure, with many vendors seeing low software quality as a viable business model.

Testing failures occur regularly, but few executives connect this with diminishing skills. Testing is simply another problem to offload to third parties, and if that fails, to a different testing service provider. Meanwhile, the myth that testing is checking continues to ride high. With continued developments in model based testing and test driven development, many conclude that “the testing problem” has finally been solved by developers.

At the same time, growth in user testing and SME testing further erodes territory that formerly belonged to the tester. Eventually, The International Institute of Business Analysts – who already claim “validation” as part of their body of knowledge – subsume any testing not performed by the developer or end user. Testers find themselves caught in a pincer from which there is no escape. In one form or another testing might live on, but testing as a profession is dead.

Scenario 2: Islands of Hope

The commoditization effect continues unabated, largely conquering the enterprise. Whilst this also spreads to software vendors, some see quality as a means to differentiate themselves in a marketplace largely defined by buggy software and long waits in the IVR-hell of technical support.

Silver bullets come and go, each long on promise and short on delivery: none can resolve “the testing problem”. Whilst successful software development is best served by skilled people working together, that answer is too hard for most: many executives reach to standards and process compliance in the hope of a solution. This only accelerates the decline of tester skills.

However, a scattered handful of executives recognize this problem. Perhaps they were testers themselves, have been fortunate enough to work with skilled testers, or simply see things clearly. These individuals cluster, seeking the company of the like minded, gravitating to those organizations that differentiate themselves on quality. In this way a market niche is born, one in which a small number of skilled testers not only survive, but thrive.

Scenario 3: Generational Change

The commoditization of testing continues for years, driven by a generation of IT managers who never witnessed the power of skilled testing.

The ranks of unskilled testers swell. Lacking a demand for skill, few employers are interested in their development. Yet this cannot suppress the human appetite to learn; many testers turn to the Internet in order to develop themselves, and the Internet is owned by the passionate.

Online coaching and training – much of it voluntary – flourishes. Soon, supply outgrows the diminishing demand for skill. Many testers drop out of testing rather than serve as commodity checkers. Some leave the industry altogether, many become developers or project managers. As demographics works its magic, those who believe in a better way find themselves in executive roles. They demand testers with skill; they demand that testing be at the heart of software development. The feedback loop driving commoditization slows and finally reverses as a paradigm shift begins to take hold.

I don’t have a crystal ball, and these scenarios have no real predictive power. However, through the exploration of scenarios such as these, we can gain insights as to how to navigate the future – both as individuals and as a craft. I invite you to add your own.


The Commoditization of Testing

As testers, one of the greatest challenges we face is the perception that the service we provide is a commodity. This may not be universal, but it is endemic in many parts of the IT industry.

Is software testing a commodity? According to Investopedia; “When a product becomes indistinguishable from others like it and consumers buy on price alone, it becomes a commodity”. In order for testing to qualify as a commodity, testing services must lack qualitative differentiation. Yet testing has many sources of differentiation:

  • Approach. Testing is context dependent. Some testers will bring practices and experiences that are relevant to a project, others will not. Some will attempt to bend project context to their approach, others will adapt to its specific needs. Perhaps one day this will cease to be a source of differentiation, but not today.
  • Location. Testing involves the discovery, trade and transformation of information. When communication channels are open and information is shared, remote –even very remote- testing is feasible. When information is hidden and hallway decisions are the norm, colocation is a huge advantage. Project politics and project culture can have a significant impact on the value provided by testers in different locations.
  • Skills. Testing is a vast and growing discipline, with ample room for specialization. Again, context plays a role: different projects will require testers with different skills. Further, the range of skills required is mind boggling: modeling skills, analysis skills, design skills, communication, relationship and interpersonal skills. Indeed, testing thrives on variety – the range of insights that different people with varied skills, outlooks and experiences bring to bear. The notion that testing is undifferentiated is anathema to testing itself.

In short, not all testing services are equal, nor should they be. Testing is not a commodity.

Then where does this perception come from? It originates from the belief that testing is unskilled. The most obvious act in testing is checking, which in itself requires little skill. In contrast, skilled activities such as modeling, analysis, design and evaluation are harder to observe. This belief is as superficial as believing that the cover is all that makes up a book.

Why is this important? The view that testing is a commodity provided by the unskilled has chilling effects on collaboration, effects that can destroy the mutual interdependence between testers and project.

Let’s consider one possible scenario that illustrates how this can play out. Imagine for a moment that you are a non-testing participant on a project.  Imagine that you see testing as essentially unskilled, a simple matter of translating specifications into scripts and then executing them.

You’re a busy person, and like anyone else on the project you are probably juggling a dozen different things at any given moment. The testers keep calling you, or emailing you, or messaging you, or asking for meetings. Why do they keep bothering you?!? Can’t they just get on with scripting? Can’t they do their jobs?

Testing depends on information. Mere specifications are a remarkably low bandwidth form of communication and on their own are seldom adequate for effective testing. Testers need to ask questions, it’s what they do. Yet by doing so, the tester can fail in the eyes of their stakeholders.

This is getting crazy, the testers won’t give up. Now they are calling the PM, BA and developers. Don’t they know we’ve got deadlines? Don’t they know we’ve got enough of our own work to do, without having to do theirs as well? It’s time to channel this through a handful of people.

A common reaction to the “we’re too busy to answer questions from testers” problem is to enact a communication plan which restricts the flow of information to (and from) testers through a small group of people. Those people may not have direct access to the relevant information, so may in turn have to ask other members of the project, thus continuing the distraction and introducing rounds of questions, answers and clarifications bouncing back and forth. With communications bandwidth restricted, and telephone game distortion, the value of information available to the testers diminishes rapidly.

What’s with these testers anyway? Not only are they a nuisance, but now I’ve seen their scripts, they’re missing important stuff! And that bug that got through the other week…this can’t go on. The testers are clearly rubbish, we need to manage them the directly.

At this point, the relationship between the testers and the wider project has completely failed. It is almost inevitable that this will lead to micromanagement, a master/slave relationship that further inhibits the testers’ opportunities for success. The mechanisms for this are varied. Perhaps the absence of trust will lead to demands for excessive documentation. Perhaps process or standards compliance will be enforced, in word rather than in spirit. Perhaps unsound performance metrics will be imposed. Each introduces opportunity costs, degrading the volume and value of the information the testers can provide. Each further distorts tester behavior. Each will increase the real cost of testing. And satisfaction in the testers will spiral ever downwards until the project comes to its naturally end, or the testers are dismissed.

A further implication of commodity testing is the notion that testing services are essentially fungible, i.e. that it does not matter who provides the service, or from where. All that matters is price; this is labor arbitrage. The result is a race to the bottom, with testing service providers in the lowest cost regions of the world the front runner. Remote testing may be successful in some situations, but imagine the scenario above compounded with multiple time zones and cultural factors such as power distance. In such circumstances, success is not an option.

How might things be different? We testers need to be aware of these issues, and to realize that things can be different. We need to continuously develop our skills, to market the importance of them, and to find ways to give visibility to those aspects of testing that are not checking. The alternatives to our craft are unthinkable.

It’s About The Testing, Stupid

By now you are most likely aware of the drama unfolding in the context driven community:

In some ways, I am saddened by this schism. Context driven testing means a lot to me, and I owe it a huge debt:
  • it taught me to question my context, instead of making assumptions
  • it taught me to think, instead of blindly assuming the best way
  • it taught me to be aware of my own biases, instead of letting them blind me.

And yet, as context driven testers go, I am a heretic: I am a member of the Canadian Software Testing Board and support certification as a means of promoting tester education. In the context driven world, “certification is bad” is a sacred cow, yet I found the ISTQB syllabi to be of great value in my own development. It helped to open my eyes to the breadth of testing and served as a fantastic jumping off point to a wide range of related literature.

Does that mean I believe in best practice? No, I believe in context driven selection of practices that will work.
Does that mean I believe that certifications are the best and only way to educate testers? Absolutely not. Indeed, I am deeply impressed by the depth and challenges of BBST, and have signed up as a volunteer instructor. That, and I’m looking forward to attending RST later this year.
Ironically, these views make me a heretic from the perspective of other “schools”, and my self-identification as a context driven tester has caused some eyebrows to raise in suspicion.
I shouldn’t have to write in these terms; about schisms, heresy and sacred cows. It may be human nature to form cliques with similar norms and values, and to exclude those who don’t conform – but does denying diversity serve the goal of improving testing?
I am passionate about testing, about providing the best testing I possibly can, and about helping others do so. It’s all about the testing: I’ll take from any source that can help me with these goals. Context driven testing helped to open my mind, and I’ll willingly trade ideas with anyone – from any “school” – who genuinely wishes to do so.
Context driven testing may not be a religion, but perhaps it can evolve into a broad church for those who care deeply about testing, and who are willing to step beyond pronouncements about the ONE right way to test.