Following an excellent presentation by James Christie at CAST2014, I participated in drafting a petition calling on the International Organization for Standardization (ISO) to withdraw those parts of the ISO 29119 standard for software testing that have been issued to date, and to suspend production of the remaining parts.
You can find the petition here: http://www.ipetitions.com/petition/stop29119, and this is why you should sign it.
A Warning From History
In 1979, with support from the Thatcher government, the British Standards Institution (BSI) first published the BS 5750 series of standards. These made provision for organizations to become “certified” and display a mark of registration, supposedly a sign of quality. By 1987, the British Government had convinced the International Organization for Standardization (ISO) to adopt BS 5750 as the basis for an international set of standards: ISO 9000.
Adoption was rapid. In the UK this was driven by Department of Trade and Industry (DTI) grants to firms who chose to register, regulation (e.g. the 1993 adoption of ISO 9001 conformance as a requirement of Oftel’s Metering and Billing scheme), and market coercion: the threat that large purchasers would only source goods and service from suppliers who were registered. Similar patterns emerged internationally, with many organizations being convinced that EU adoption of the standard meant that registration was a cost of doing business in Europe. By 2009 over a million firms were registered worldwide: over a million firms supporting an ecosystem of consultants, training providers, and assessors.
Nothing so dramatic has yet been seen in the world of software testing. When compared to the uptake of ISO 9000, IEEE 829, BS 7925 and their ilk seem to have been largely ignored. No wonder many testers seem to view the new ISO 29119 series of software testing standards with apathy.
Yet ISO 29119 could be the ISO 9000 of software testing. Whilst other testing standards were relatively limited in scope (documentation, component testing etc.) 29119’s stated aim is to “define [a] set of standards…that can be used by any organization when performing any form of software testing” (ISO/IEC/IEEE 29119-1:2013, Introduction). Any form of testing, in any organization, in any context. That means YOU.
Further, ISO 29119 is designed to support conformance and registration: conformance may be claimed to parts 2, 3 and 4, ISO 29119-2 describes how full or partial conformance with its processes might be achieved, the website of the ISO working group responsible for the standards describes how ISO/IEC 33063 would be used to assess test processes against ISO 29119-2. ISO 29119 has all the makings of a registration regime that would extend to any testing, anywhere. The bonfire is set; all that is needed is a spark to light it.
In the late 70’s, such a spark came in the form of Thatcher’s desire to repeat the “Japanese miracle” and reinvent British Management. What might ignite ISO 29119? Take banking, with it’s explosion in regulation following the last crash. Now imagine an event, a software failure, a Flash Crash or Knight Capital writ large, something that causes significant economic damage. Or take an airline, with hundreds of souls aboard. Or drug production. Or any of dozens of examples: software is everywhere. Now imagine the outcry: “No Senator, despite the existence of internationally agreed standards, we did not apply those standards to our testing”1.
It’s not terribly hard to imagine, indeed it may already be happening: some vendors are calling for the adoption of standards as a result of the Healthcare.gov fiasco2. Love it or loathe it, the one thing you cannot afford to do is ignore ISO29119.
A Flawed Approach
Standards in manufacturing make sense: the variability between two different widgets of the same type should be minimal, so acting in the same way each time a widget is produced is desirable. This does not apply to services, where demand is highly variable, or indeed in software, where every instance of demand is unique.
Attempting to act in a standardized manner in the face of variable demand is an act of insanity: it’s akin to being asked to solve a number of different problems yet merrily reciting the same answer over and over. Sometimes you’ll be right, sometimes wrong, sometimes you’ll score a partial hit. In this way, applying the processes and techniques of ISO 29119 will result in effort being expended on activities that do nothing to aid the cause of testing.
And in testing, that’s a major problem. When we test, we do so with a purpose: to discover and share information related to the quality. Any activity, any effort that doesn’t contribute to doing so is waste. As “complete” testing is impossible, all testing is a sample. Any such waste results in a reduction in sample size, it equates to opportunity cost: an opportunity lost to perform certain tests. For a project constrained by quality, this translates into increased time and cost. For a project constrained by time or money, this translates into a reduction in the information available to stakeholders, and a corresponding increase in risks to quality.
The 29119 crowd might tell you that the new standard takes this into account, that it encourages you to tailor your application of the standard to each project, that it is sufficiently comprehensive that you need only select the processes and techniques that apply. This is the Swiss Army Knife fallacy: if you have one you’ll never need another tool. One of the problems with a Swiss Army knife is that it’s not much use if you need a pneumatic drill, or an ocean liner.
Training testers to use a standard in this way has a tendency of framing their thinking. Rather than trying to solve testing problems, they instead seek to choose from a set of ready-made solutions that may or may not fit. I once conducted a highly informal experiment with two groups of students. The first group was trained in a set of formal test design techniques. The second received a short briefing on some general testing principles and the use of the heuristic test strategy model3. Both were then tasked with creating a set of test ideas for the same product. The difference between the ideas generated by the two groups was stark. The first group came up with a predictable set of equivalence classes etc., whilst the second group came up with a rich and varied set of ideas. When released, and if widely adopted, part 4 (on test techniques) will give rise to a generation of testers locked firmly inside the 29119 box, without the ability or freedom to solve the problems they need to solve.
And that’s not the worst of it. For my sins, I spent a number of years as an ISO9000 auditor. It seemed like a great idea at the time: understand the system, monitor the system, improve the system. Gradually, I realized this wasn’t the reality. People were documenting the system, documenting their reviews, documenting their responses to audit findings, and doing very little by way of improving the operation of the business. What the hell was going on? Goal displacement, that’s what. We’d created a machine geared towards complying with the standard, demonstrating conformance to the satisfaction of an assessor, and maintaining our registration once obtained. Somewhere along the line, the goal of improving our business had been forgotten. This phenomena isn’t limited to organizations that seek compliance with external standards. Not so very long ago, I watched an organization doing much the same whilst attempting to standardize their testing processes. Significant effort was directed to completing templates for test documentation, reporting metrics and self-assessing vs. the internal standard – all with no regard for the relevance or value of doing so for the projects that testing was meant to be serving.
Waste, waste, and more waste.
So when standards proponents tell you that following ISO 29119 will improve the efficiency or effectiveness of your processes, call them out: far from making testing more efficient or effective, conformance will have the opposite effect.
The text of ISO 29119 claims that it is “an internationally-agreed set of standards for software testing”. This agreement is meant to be the product of consensus, defined by ISO as “general agreement, characterized by the absence of sustained opposition to substantial issues by any important part of the concerned interests and by a process that involves seeking to take into account the views of all parties concerned and to reconcile any conflicting arguments” (ISO/IEC Guide 2:2004).
There is no such consensus. Instead there is a small group of members of a working group who claim to represent you. Meanwhile, hundreds of testers are calling for the withdrawal of ISO 29119.
There is no consensus; there will be sustained opposition. Join the opposition: http://www.ipetitions.com/petition/stop29119
1 For more on this theme, read “Uncle Bob” Martin’s After the Disaster
2 My thanks to James Christie for drawing attention to this during his CAST 2014 presentation