Category Archives: Opinion


Party like it’s 1979

We are rolling back the clock so as to prevent

you from finding better ways to test software.

Through this work we will make you value:


Management control over individual accountability

Documentation over finding out about software

Policing the lifecycle over collaboration with the team

Detailed test planning over exploration and discovery


That is, we can’t even begin to imagine

how you might have come to value the things on the right.

-The International Organization for Standardization

Stop 29119

Following an excellent presentation by James Christie at CAST2014, I participated in drafting a petition calling on the International Organization for Standardization (ISO) to withdraw those parts of the ISO 29119 standard for software testing that have been issued to date, and to suspend production of the remaining parts.

You can find the petition here:, and this is why you should sign it.

A Warning From History

In 1979, with support from the Thatcher government, the British Standards Institution (BSI) first published the BS 5750 series of standards. These made provision for organizations to become “certified” and display a mark of registration, supposedly a sign of quality. By 1987, the British Government had convinced the International Organization for Standardization (ISO) to adopt BS 5750 as the basis for an international set of standards: ISO 9000.

Adoption was rapid. In the UK this was driven by Department of Trade and Industry (DTI) grants to firms who chose to register, regulation (e.g. the 1993 adoption of ISO 9001 conformance as a requirement of Oftel’s Metering and Billing scheme), and market coercion: the threat that large purchasers would only source goods and service from suppliers who were registered. Similar patterns emerged internationally, with many organizations being convinced that EU adoption of the standard meant that registration was a cost of doing business in Europe. By 2009 over a million firms were registered worldwide: over a million firms supporting an ecosystem of consultants, training providers, and assessors.

Nothing so dramatic has yet been seen in the world of software testing. When compared to the uptake of ISO 9000, IEEE 829, BS 7925 and their ilk seem to have been largely ignored. No wonder many testers seem to view the new ISO 29119 series of software testing standards with apathy.

Yet ISO 29119 could be the ISO 9000 of software testing. Whilst other testing standards were relatively limited in scope (documentation, component testing etc.) 29119’s stated aim is to “define [a] set of standards…that can be used by any organization when performing any form of software testing” (ISO/IEC/IEEE 29119-1:2013, Introduction). Any form of testing, in any organization, in any context. That means YOU.

Further, ISO 29119 is designed to support conformance and registration: conformance may be claimed to parts 2, 3 and 4, ISO 29119-2 describes how full or partial conformance with its processes might be achieved, the website of the ISO working group responsible for the standards describes how ISO/IEC 33063 would be used to assess test processes against ISO 29119-2. ISO 29119 has all the makings of a registration regime that would extend to any testing, anywhere. The bonfire is set; all that is needed is a spark to light it.

In the late 70’s, such a spark came in the form of Thatcher’s desire to repeat the “Japanese miracle” and reinvent British Management. What might ignite ISO 29119? Take banking, with it’s explosion in regulation following the last crash. Now imagine an event, a software failure, a Flash Crash or Knight Capital writ large, something that causes significant economic damage. Or take an airline, with hundreds of souls aboard. Or drug production. Or any of dozens of examples: software is everywhere. Now imagine the outcry: “No Senator, despite the existence of internationally agreed standards, we did not apply those standards to our testing1.

It’s not terribly hard to imagine, indeed it may already be happening: some vendors are calling for the adoption of standards as a result of the fiasco2. Love it or loathe it, the one thing you cannot afford to do is ignore ISO29119.

A Flawed Approach

Standards in manufacturing make sense: the variability between two different widgets of the same type should be minimal, so acting in the same way each time a widget is produced is desirable. This does not apply to services, where demand is highly variable, or indeed in software, where every instance of demand is unique.

Attempting to act in a standardized manner in the face of variable demand is an act of insanity: it’s akin to being asked to solve a number of different problems yet merrily reciting the same answer over and over. Sometimes you’ll be right, sometimes wrong, sometimes you’ll score a partial hit. In this way, applying the processes and techniques of ISO 29119 will result in effort being expended on activities that do nothing to aid the cause of testing.

And in testing, that’s a major problem. When we test, we do so with a purpose: to discover and share information related to the quality. Any activity, any effort that doesn’t contribute to doing so is waste. As “complete” testing is impossible, all testing is a sample. Any such waste results in a reduction in sample size, it equates to opportunity cost: an opportunity lost to perform certain tests. For a project constrained by quality, this translates into increased time and cost. For a project constrained by time or money, this translates into a reduction in the information available to stakeholders, and a corresponding increase in risks to quality.

The 29119 crowd might tell you that the new standard takes this into account, that it encourages you to tailor your application of the standard to each project, that it is sufficiently comprehensive that you need only select the processes and techniques that apply. This is the Swiss Army Knife fallacy: if you have one you’ll never need another tool. One of the problems with a Swiss Army knife is that it’s not much use if you need a pneumatic drill, or an ocean liner.

Training testers to use a standard in this way has a tendency of framing their thinking. Rather than trying to solve testing problems, they instead seek to choose from a set of ready-made solutions that may or may not fit. I once conducted a highly informal experiment with two groups of students. The first group was trained in a set of formal test design techniques. The second received a short briefing on some general testing principles and the use of the heuristic test strategy model3. Both were then tasked with creating a set of test ideas for the same product. The difference between the ideas generated by the two groups was stark. The first group came up with a predictable set of equivalence classes etc., whilst the second group came up with a rich and varied set of ideas. When released, and if widely adopted, part 4 (on test techniques) will give rise to a generation of testers locked firmly inside the 29119 box, without the ability or freedom to solve the problems they need to solve.

And that’s not the worst of it. For my sins, I spent a number of years as an ISO9000 auditor. It seemed like a great idea at the time: understand the system, monitor the system, improve the system. Gradually, I realized this wasn’t the reality.  People were documenting the system, documenting their reviews, documenting their responses to audit findings, and doing very little by way of improving the operation of the business. What the hell was going on? Goal displacement, that’s what. We’d created a machine geared towards complying with the standard, demonstrating conformance to the satisfaction of an assessor, and maintaining our registration once obtained. Somewhere along the line, the goal of improving our business had been forgotten. This phenomena isn’t limited to organizations that seek compliance with external standards. Not so very long ago, I watched an organization doing much the same whilst attempting to standardize their testing processes. Significant effort was directed to completing templates for test documentation, reporting metrics and self-assessing vs. the internal standard – all with no regard for the relevance or value of doing so for the projects that testing was meant to be serving.

Waste, waste, and more waste.

So when standards proponents tell you that following ISO 29119 will improve the efficiency or effectiveness of your processes, call them out: far from making testing more efficient or effective, conformance will have the opposite effect.

No Consensus

The text of ISO 29119 claims that it is “an internationally-agreed set of standards for software testing”. This agreement is meant to be the product of consensus, defined by ISO as “general agreement, characterized by the absence of sustained opposition to substantial issues by any important part of the concerned interests and by a process that involves seeking to take into account the views of all parties concerned and to reconcile any conflicting arguments” (ISO/IEC Guide 2:2004).

There is no such consensus. Instead there is a small group of members of a working group who claim to represent you. Meanwhile, hundreds of testers are calling for the withdrawal of ISO 29119.

There is no consensus; there will be sustained opposition. Join the opposition:



1 For more on this theme, read “Uncle Bob” Martin’s After the Disaster

2 My thanks to James Christie for drawing attention to this during his CAST 2014 presentation



Checking Is Not Evil

A few days ago, over at, Cem Kaner expressed the view that manual exploratory testing had become the new Best Practice. Cem’s comments resonate strongly for me as I have recently recognized and sought to correct that in my own thinking.

Whilst on BBST Test Design last year, Cem called me out on one question where I’d answered from a position of context driven dogma rather than any kind of well thought out position. This was an invaluable lesson that made me begin to question my prejudices when selecting a test approach.

I recognized that I had begun to show a strong preference towards manual exploration over automated testing, of disconfirmatory testing over confirmatory checking. I resolved to challenge myself, and for every approach I considered, to think of at least two others so as to compare and contrast their relative merits.
Then came the moment of truth: a new project. And as luck would have it, one that would go to the very heart of this issue.
The project in question is all about data. There are many thousands of conditions that are specified detail, the data combinations expand this out to potentially billions of individual checks. I would be ignoring a significant part of my testing mission were I to disregard either checking or automation in favor of manual ET. In short, the context demands that large-scale mechanized checking form a big part of the test approach.
Does this mean that we will rely solely on checking and disregard exploration?
Does a check-heavy strategy make this any less context driven?
No, and no. As I wrote here, checking may be necessary but it is not sufficient. This project is not unique: there is plenty of opportunity for unfulfilled needs, undesired or unexpected behaviour. We will test the specifications before a single line of executable code is available. We will challenge our own understanding of the design, and seek to align the perspectives and understanding of others. We will use risk based test design and seek to reveal failures. We will accept that the specification is not perfect, and that we will need to adjust our thinking as the project evolves. We may check, we may not test manually, but the essence of what we do will remain both exploratory and context driven.
Checking is not evil, nor (as far as I am aware) did anyone say that it is. 

It’s About The Testing, Stupid

By now you are most likely aware of the drama unfolding in the context driven community:

In some ways, I am saddened by this schism. Context driven testing means a lot to me, and I owe it a huge debt:
  • it taught me to question my context, instead of making assumptions
  • it taught me to think, instead of blindly assuming the best way
  • it taught me to be aware of my own biases, instead of letting them blind me.

And yet, as context driven testers go, I am a heretic: I am a member of the Canadian Software Testing Board and support certification as a means of promoting tester education. In the context driven world, “certification is bad” is a sacred cow, yet I found the ISTQB syllabi to be of great value in my own development. It helped to open my eyes to the breadth of testing and served as a fantastic jumping off point to a wide range of related literature.

Does that mean I believe in best practice? No, I believe in context driven selection of practices that will work.
Does that mean I believe that certifications are the best and only way to educate testers? Absolutely not. Indeed, I am deeply impressed by the depth and challenges of BBST, and have signed up as a volunteer instructor. That, and I’m looking forward to attending RST later this year.
Ironically, these views make me a heretic from the perspective of other “schools”, and my self-identification as a context driven tester has caused some eyebrows to raise in suspicion.
I shouldn’t have to write in these terms; about schisms, heresy and sacred cows. It may be human nature to form cliques with similar norms and values, and to exclude those who don’t conform – but does denying diversity serve the goal of improving testing?
I am passionate about testing, about providing the best testing I possibly can, and about helping others do so. It’s all about the testing: I’ll take from any source that can help me with these goals. Context driven testing helped to open my mind, and I’ll willingly trade ideas with anyone – from any “school” – who genuinely wishes to do so.
Context driven testing may not be a religion, but perhaps it can evolve into a broad church for those who care deeply about testing, and who are willing to step beyond pronouncements about the ONE right way to test.

Should I Start Testing?

I recently participated in a Software Testing Club discussion about Quality Gates. This led me to reflect a little more on the subject.

My first test management gig was running system integration testing for an enterprise project. The phase was planned to last for five weeks. System testing for a number of component systems was running late, and that was making the project look bad. The project manager impressed on me the political importance of starting system integration on time.

“System test should be done in another week or so” she said.

“So you should be able to catch that up in the last month of integration”. I reluctantly agreed.

Twenty five weeks of hell later, I submitted the exit report. Nothing had worked out as expected. Delays had continued on completion of some systems. Others that were supposedly complete either had gaping holes where functionality should have been, or turned into a bugfest. Attempting to test and raise system integration level bugs simply poured fuel on the fire, adding to the confusion.  I swore that from that point on I’d only test software that was ready to be tested: that I’d implement strict quality gates and hard entry criteria.

I can understand why Quality Gates have become an article of faith to many testers; the above experience led me in the same direction. Fortunately, experience has also given me a reality check.

Let’s roll the clock forward a few years and consider another project: one on which time to market was the most critical factor.  The end date was not going to change: bugs or no bugs. Development was running late, and enforcing entry criteria would almost guarantee that no testing would be performed by my team, no feedback would be provided to development, no bugs would be found or fixed. I abandoned my gates, and helped to identify some major issues that we were able to iron out before we shipped.

In the first example, quality gates might have helped me. In the second, they were utterly inappropriate. The problem with quality gates goes deeper than a discussion of relevance to different contexts however: they are a trivial solution to a complex problem. They seek to reduce the process of answering the question “should I start testing” to simple box checking. Perhaps the most insidious problem with gates is that they tend to emphasize reasons NOT to test. Thus the tester who has come to believe that testing is like a sewerwhat you get out of it depends on what you put in to it * – can use them as an excuse not to test. Unit testing incomplete? Don’t test. Major bugs outstanding? Don’t test. Documentation outstanding? Don’t test. All these factors miss a vital point: reasons TO test.

Even when test entry criteria have not been fulfilled, there are many good reasons to test:

  • Perhaps you can provide early feedback on items that aren’t quite ready for the big time
  • Perhaps you can learn something new and think of new tests
  • Perhaps you can become more familiar with the software
  • Perhaps you can test your own models and assumptions about the software
  • Perhaps you test whether your test strategy is workable.

The decision to start testing demands more thought than simple box checking can provide. Instead, questions like these can serve as a guide:

  • If I we’re to start testing now, what kind of constraints would my testing be under?
  • Given those constraints, what testing missions could I accomplish?
  • For those missions, what kind of value could I provide to the project?
  • What is the value of tasks that I might forego in order to test now?
  • If I were to test now, what other costs might this impose on the project?

Next time you decide whether or not to test, use your brain, not a check list.

*A nod to Tom Lehrer.


Counting Experience

Once upon a time, I believed that only testers should test, and testing experience counted for everything.  This was an easy trap to fall into: after all, I had been around the block. I’d learned lots about testing approaches, different strategies, methodologies and techniques. Looking back at my earlier efforts, and many mistakes, it was easy to think “if only I knew then what I know now!” and ascribe any improvement to experience.

Dumb. Arrogant. Mistake.

What changed my mind?

Let me briefly describe the project: we were customizing an enterprise solution owned by Client A for use by Client B. Due to the complexity of the product, the lack of any models or documentation, and the significance of the changes, this would be no simple matter. In addition, time scales were tight and we were iterating rapidly. I chose a testing approach that relied heavily on exploratory testing, with automation support for those areas that were considered high regression risk or eminently suitable for data driving.

The exploratory testers were to be the vanguard: first to test new code, first to test fixes, first to investigate and qualify bugs reported by others. This was to be the most critical testing role on the project. For this I was given two subject matter experts seconded from Client A: testers whose sole testing experience was a little bit of UAT.

Now, there’s a common misperception that I often hear about ET: “you need a lot of testing experience to do that”. Perhaps I could have listened to this conventional wisdom. Perhaps I could have succumbed to my prejudices. After meeting the testers in question, I played a hunch and chose not to.

We got started with some basics: discussing the impossibility of complete testing, the oracle problem, bug reporting expectations and the overall approach to testing for the project. Then we got testing. To demonstrate the mechanics of SBTM, I led the first couple of test sessions, after which I dropped back to chartering and reviewing session reports. Within a few weeks I pretty much left them to charter themselves, with a daily huddle to discuss and prioritize things to test.

In the early days I monitored progress intensively:

  • When I heard them debate whether a particular behaviour was a bug, I interrupted with a brief breakout session to discuss the relationship of quality and value, and a variety of different oracles they might consider.
  • When I heard them struggling with how to test a particular feature, I’d introduce them to a few different test design techniques and heuristics that might be relevant.
  • I’d review bug reports, and provide feedback as to follow-up testing they should consider.

Pretty soon they were a wonder to behold. They quickly assimilated every idea and concept thrown at them. The bugs they identified demonstrated significant insight into the product, its purpose, and the needs of its eventual users. It was fascinating to listen to animated discussions along these lines:

Tester 1: Is this a bug?

Tester 2: Hmm, I don’t think so. It’s consistent with both the requirements and the previous version.

Tester 1: But what if [blah, blah, blah]…doesn’t that really kill the value of this other feature?

Tester 2: Got you. That could be a problem with the requirements; we need to talk to the BA.

This pair had rapidly become the most impressive testing duo I’d ever had the pleasure of working with. How had this happened? They brought with them a blend of aptitudes and experiences that far outweighed their relative lack of testing experience:

  • They had open minds, a willingness to learn, and no preconceived notions as to “the one true way” to test.
  • They exhibited an insatiable curiosity: a desire to understand what the software was doing and why.
  • Having lived with the previous version of the product, they were dedicated to delivering quality to other users like them.
  • Their experience as users meant that they had well refined internal oracles that gave them insight into what would diminish user value.

Their experience counted for a lot, but not their testing experience: any testing know-how they needed they were able to learn along the way.

I’m not claiming that testing experience counts for nothing: I’ve also worked in situations where testers needed significant experience in testing, and specific types of testing, or else be eaten alive by a sophisticated client. Testing experience is only one piece in a complicated puzzle that also includes domain experience, technology experience, attitude and aptitude.  Different situations will demand a different blend.

None of these factors are simple. Consider testing experience: this is not a straightforward, indivisible commodity. Testing is a diverse field, and highly context dependent. What works for you in your context might not work for me in mine. Often the recruitment of testers boils down to a simple count of years in testing. A tester can spend decades testing in one context, but when moved suffer from “that’s not how you test” syndrome.  Such an individual is ill equipped to learn or even consider the approaches and techniques that are relevant to a new situation. Even a testing virgin could be a better choice, if accompanied by an open mind. Diversity of experience, and a willingness to learn and adapt, count for far more than years. Counting experience is for fools.

Babies 0: Bathwater 1

Michael Bolton took me a little by surprise this week when he tweeted a link to my blog with the #agile hash tag: I consider myself about as agile as a two-by-four.

That’s not to say that I am an agile virgin: I have tested on a few agile projects, and managed testing teams supporting iterative development.  I don’t however consider myself an expert, having cut my teeth on the waterfall test crunch of doom.

The best way that I can think of to characterize my experiences with agile is with one word: bipolar.

Let’s take the good first (call it Project 1), and describe it in terms of the Agile Manifesto:

  • Individuals and interactions (constant BA/dev/test interaction) over processes and tools (lots of emphasis on process and tools, constantly being tweaked and improved, more than I’ve ever witnessed elsewhere – is this some kind of agile paradox?).
  • Working software (every iteration gave us something that worked) over comprehensive documentation (minimal, but that which existed was both useful and used – i.e. no dust gatherers).
  • Customer collaboration (frequent engagement in refining requirements, defining acceptance criteria and acceptance testing) over contract negotiation (minimal).
  • Responding to change (effectively managed through backlog) over following a plan (burn down estimation).

This was a positive experience, the team gelled, the software hung together nicely and the customer left happy.

Now let’s consider the polar opposite (Project 2):

  • Individuals and interactions (none) over processes and tools (none).
  • Working software (take a guess…) over comprehensive documentation (what documentation?).
  • Customer collaboration (who?) over contract negotiation (what?).
  • Responding to change (hourly) over following a plan (plan is a four letter word).

This project called itself agile, assumed that meant “Process – Pah! Documentation, we don’t need no stinking documentation!” and put nothing in their place.  In short, the lunatics were running the asylum – and I’m not talking developers here – this was the whole team approach to insanity.

The funny thing is, I’ve been on waterfall projects that strongly resembled Project 1: good interaction, sharing of information, useful documentation, and customer collaboration.  I’ve also been on waterfall projects that could easily have been confused with Project 2: madness personified.  The common denominator is not the label, it is the team dynamics.

Software engineering is, when all things are said and done, a social activity.  We’re not making widgets here, there is no physical product being passed from A to B: our products are information, ideas and words – and these are like the wind. Without good collaboration the moment is lost and the information along with it. Process and documentation can at least provide some base level of information sharing. Rip these out without replacing them with people talking to one another and the baby has gone out with the bathwater. Regardless of methodology and other labels: effective sharing of information helps teams to succeed.

Whatever your methodological preferences, please look after your babies.