I’ve been noodling with this post for a while now, never quite finishing it off, and then yesterday Dorothy Graham posted Is it dangerous to measure ROI for test automation?, spurring my reply: Yes Dot, yes it is.

All too often I’ve seen ROI used in attempt to either sell or justify test automation. The reasoning goes like this: replace a bunch of manual tests with automated ones, it’ll save money. This is an argument based on a great big stinking ROLie. Automated checks are never a replacement for tests conducted by people. Tests and automated checks are not substitute goods. Machines cannot observe anything other than that which they’ve been programmed to check, their judgment is limited to the algorithms with which they have been provided, they are incapable of insight or intuitive leaps, and they can never play a hunch.

Even when used honestly, as a sanity check of investment decisions, this kind of thinking is perverse. As Dot states: “If we justify automation ONLY in terms of reduced human effort, we run the risk of implying that the tools can replace the people.” In other words, we can perpetuate the myth that testing is a commodity made up of low skilled and easily replaceable parts.

I do not believe that R.O.I. is entirely irredeemable. Such cost comparisons can make sense, but only if applied below and across tests, at the level of testing tasks. For example, earlier this year, I needed to make some tooling decisions relating to the testing of a large volume and highly complex ETL process. Rather than evaluating which tests could be replaced with automation, I instead looked at which tasks should be performed by people, and which should be performed by a machine. Here’s a thumbnail sketch of the reasoning:

  • Data mining, repetitive application of rules: automate.
  • Data conditioning: conceptually possible to automate but insanely expensive to do so exhaustively: let’s stick with people.
  • Expected result generation, high volume repetitive application of rules: automate.
  • Result reconciliation, literally billions of checks per data load: automate.
  • Bug isolation and reporting, investigation and judgment required: need people.

Of course, other things factored into the decision making process (I’ll discuss that a little more at CAST 2012 and in a subsequent post), but having realized that I was beginning to lean heavily towards using large scale mechanized checking to assist with much of our testing, I wanted to carefully check my thinking. In this case, rather than seeking justification, I needed wanted to answer one simple question: Am I making idiotic use of my customer’s money? ROI served as a test of this aspect of my strategy.

Now, this is a narrow example. The parallel nature of ETL test execution lends itself to breaking out and batching the tasks that make up individual tests. For many types of testing this is impossible, and ROI useless. We need a different paradigm, a focus on value instead of replacement. This is fairly straightforward. There are many ways in which tooling can add value: it can enable testers to obtain information that would be inconceivable without tools, it can improve the accuracy and precision of information that they can access and it can enable them to provide information faster or more cost effectively. The tricky part is putting a price on that value so as to determine if a particular automation effort is a worthwhile investment. So why not simply ask? Why not simply discuss the likely value with your customer, and ask what price they would put on it? For example:

  • “This part of the application has a high number of permutations and we can only scratch the surface without automation. What price would you put on being able to identify some major problems in there? What price would you put on knowing that despite our best efforts we haven’t found any major problems with it?”
  • “The current regression checks take about a week to complete. What price would you put on being able to complete them within a few hours of a new build?”
  • “Using flags and stopwatches, we can only scale the performance tests to around 100 testers. What price would you put on a more realistic simulation?”

This might lack the appearance of objectivity that accompanies ROI, but let’s face it; the typical ROI is so speculative and riddled with estimating error as to be laughable. What this approach provides is a quick and easy way of getting to the business of value, and focusing on what tooling can do for our customers rather than only what it will cost them.

6 thoughts on “R.O.Why?”

  1. Hi Iain,

    Thanks for responding to my blog post – a friend of mine happened to notice it and told me about it, otherwise I wouldn’t have known. I have put a link to your post in my blog now.

    Yes, ROI is often used to sell or justify test automation, and sometimes it is an “ROLie” and used as an excuse to get risk of testers and to regard machines and people as interchangeable, so that people become a commodity – you put it very well, and this was my concern too.

    I like your examples of when to automate and when not to automate, and I agree – get people to do what they do best and get computers to do what they do best.

    But ROI is sometimes needed to communicate to the high level managers in an organisation who control the “purse strings”. Good automation does not come out of a box, whether it is a commercial tool or open source, so management support is critical if test automation is to be long-lasting and wide-spread. Unrealistic expectations often lead to automation failure.

    1. Thank you Dot for the thoughtful response. I agree that setting realistic expectations is important. What if the expectation that your conversation with executives will revolve around ROI is itself unrealistic? If the value of a given tool is something other than replacement, how does ROI help to establish realistic expectations?

  2. Jonathan Kohl pointed me to this wonderful blog post, which underscores something that I had always suspected but was never able to articulate (having ignored too many offers to attend an accounting class). Like “social media”, test automation is not an investment.

    About marketing activity, Sean (a Chief Financial Officer; someone who is very well paid to understand these things) says “marketing activity is not an investment. An investment is an asset that you purchase and place on your Balance Sheet. Like an office building or a computer system. It’s something you could sell later if you didn’t need it any more. Marketing is an expense, and goes on the Profit & Loss statement.”

    He goes on: “Quick, calculate the ROI of using email within your organization. Not email marketing, but just the emails you send back and forth to get things done. Sure, you may know what it originally cost to install your email system, but how do you measure the gain achieved from it? If you are like most, you don’t know. You probably don’t care. But the absence of email in your organization would lead to more harm than good. Its ‘gain’ is not so much a measurement of return but an implicit cost of being in business.”

    I’ve always suspected that people who sell test automation based on “return on investment” didn’t understand testing. Thanks to Sean, I now grasp that they don’t understand accounting either.

    —Michael B.

    1. Thanks Michael. Jonathan linked that off Dot’s original post, and it’s a great read. Nice summary though!

  3. Although stating that “email” is not an investment in the strict sense of the Wall Street definition may be correct, we can however, put a value on the benefit of having email to an organization. It may not be easy or even worthwhile to arrive at that dollar value, but that does not preclude the fact that a price can be arrived at which compares the corporate profit of having versus not having email. ROI = profit/initial investment. So if we “bend” the rules slightly to define initial investment as the value of the corporation (in one case subtracting the upfront cost of installing corporate email), then we readily have a means for comparing ROI in each case. This avoids getting lost in the detail of semantics and instead opens up an avenue for comparisons that stakeholders may find convincing.

Leave a Reply

Your email address will not be published. Required fields are marked *