The business of testing can be fascinating: I’ve recently been grappling with some of the claims concerning different pricing models that one can apply to testing services. This short series of post explores a number of common models, reflecting my current thinking on each one.

Let’s start with the simplest and most common arrangement: Time and Materials (T&M). The vendor supplies testers at an hourly or daily rate and the client pays for the amount of testing effort that they need. Conventional wisdom suggests that this is an undesirable arrangement: pricing on the basis of inputs (i.e. effort) provides no incentive for the vendor to improve efficiency, resulting in unnecessarily high costs for the client.

I disagree, and propose that the following rule applies:

In the absence of other factors most people will try to do their best in the hope of getting more work.

With this rule in mind, a vendor has plenty of incentive to make improvements: these are a relationship building investment that can help to secure repeat business. Later, I will argue that alternative billing arrangements can sometimes interfere with this rule, but more of that to come.

Fixed Price contracts are a popular alternative to T&M: the vendor provides testing services to the client at a fixed price, generally set on a project by project basis. The client pays this price regardless of the testing effort actually expended on the project. This means that testing is not the only service that the vendor is providing: they are also accepting payment for the transfer of risks from the client. Fixed price contracts are often negotiated on the basis of T&M pricing, plus a premium to account for the risk. There’s a name for this: it’s called insurance. I’m sure that most vendors don’t realize that they’re in the insurance business, and I’ve certainly never seen any evidence of insurance expertise being applied to pricing. Many executives will tell you that the risks associated with a portfolio of such projects “net out” at a level that will be covered by the risk premium. This assumes that project delays are normally distributed with significant project delays being highly improbable. Unfortunately this kind of catastrophic delay can, and does, occur. Ever seen one? I’m guessing that you have. I’ve seen several, including one project that overran by two years at a cost of millions to the vendor. Projects that go this far off the rails can wipe out the profitability of all but the largest portfolio. Selling testing services in this way is betting against disaster for the promise of relatively modest returns.

Fixed price contracts also add to the challenge of estimation. How do you estimate how long testing will take? Many approaches to test estimation may seem plausible, rational even, yet none offers anything more than false confidence. The bottom line is that testing will last as long as testing needs to last: testers have no control over the risk tolerance clients or the quality of code, yet these are the primary factors that determine how long a project (and therefore how long testing) will take. The estimation problem, when combined with a competitive fixed price bid, leads to conflicting incentives for the vendor. On the one hand, the vendor wants to minimize risks, which can encourage them to build healthy contingency into their estimates. On the other hand, the vendor doesn’t want to price themselves out of the game, which can encourage them to bid low. Often, project management approaches are seen to be the answer: specify in detail exactly what testing “deliverables” will be provided, document explicit assumptions and if anything changes use change requests to cover the corresponding costs. This is the testing vendor’s equivalent of the insurer’s policy exclusions. The problem of this approach is that it can be extremely damaging to relationships. Try this exercise: name one popular insurance company! Whilst I’ve never personally witnessed a vendor who “low balls on price then rapes you with CRs”, I’ve heard plenty of horror stories from clients. If you are in a fixed price contract with a client, and you have any aspiration to develop a longer term relationship – or to preserve one – then tread carefully when dealing with change requests.

There are other related dysfunctions. Perceived risk breeds control, thus fixed price contracts often describe deliverables in detail. Unfortunately, deliverables are not the testing. Remember that in the discussion of T&M pricing I introduced this rule?

In the absence of other factors most people will try to do their best in the hope of getting more work.

The desire for control introduces another factor. The vendor must now balance competing desires: to do whatever is necessary to assist the project, and to mitigate their own risks through control and adherence to contracted deliverables. Excessive focus on the latter can erode the time available to investigate software and reveal new information. Ironically, this opportunity cost can prevent or delay the discovery of problems and consequently increases the probability and magnitude of overruns. This is most evident on a fixed price contract when things go wrong. When facing such a project, the vendor has an incentive to minimize their exposure by not incurring additional effort and delivering only that which they are contractually obliged to deliver. Need to explore beyond the letter of the requirements? Tough, not covered in the contract. This stick-to-the-plan-no-matter-how-screwed-it-turned-out-to-be-in-reality mentality is exactly the kind of behavior that can result in further delays to the project, reinforcing its death spiral.

Let’s return to the notion of efficiency. Does a fixed price model encourage improved efficiency over T&M? It is often suggested that it might: with a cap on the payment they will receive, the vendor should have an incentive to work in a more efficient manner. Nonsense! This is true in only the narrowest of senses: the vendor has an incentive to fulfill contracted obligations at a minimal cost, thus maximizing profit. This is financial efficiency, not testing efficiency: they are not the same thing. A vendor can improve financial efficiency through “juniorization”, the use of less expensive testers. If cheaper testers take longer to achieve similar results then it is entirely possible for such a gain in financial efficiency to be bought with a decline in testing efficiency. This is another complicating factor! The vendor now has to balance the desire to reduce costs with a desire to perform. Even if the vendor favors the latter and is able to make improvements in testing efficiency this does not automatically translate into value for the client. Testing efficiency either results in cost savings, which are not generally passed on to the client on a fixed price contract (the client pays the same regardless of the actual effort), or frees capacity that can be used to test more. This is an opportunity for a vendor to invest in the success of the project and thereby their relationship with the client. However, the control culture that sometimes permeates fixed price work can introduce resistance to the suggestion that effort be devoted to anything not covered by contracted deliverables.

Despite the risks of such performance distorting issues, a large number of testing clients are demanding fixed price contracts. And why not? To most, a transfer of risk will be significantly more important than the poorly understood and intangible effects on the testing itself. For their part, vendors – not wanting to lose business to their competitors – are accepting a business model that will cost many of them their shirts. Unfortunately, those that survive will probably ascribe their success to skill without recognizing the real reason: the good fortune to escape disastrous projects.

To be continued.