Some time ago, I was an SAP consultant. Between projects I configured a variety of demonstration systems for use in presales. These were pretty rough and ready, but they did the job. The trick (and I mean that literally) was to carefully define detailed step by step scripts, test them, and make sure that the demonstrator followed them to the letter. These provided safe pathways; routes through the application that were free of problems. A demonstrator would stray from the path at their peril; the demo would quickly fall apart if they did.
This is analogous to some User Acceptance Testing practices that I’ve observed. Do you recognize this?
The blinkered scripting scam: Acceptance tests will be scripted in advance. They will be traced to requirements. The scripts will be reviewed and pretested by the project. If all the tests pass when executed by the acceptance team, then the software will be accepted.
From a project management perspective this would seem to make sense:
- It gives the project an opportunity to check that the acceptance team is only testing behavior that is considered to be within the projects scope.
- It gives the project an opportunity to make sure that acceptance tests will pass before they are formally executed.
- It helps to ensure that the acceptance team can begin execution just as soon as the software is ready for them.
This is not testing, nor is it even meaningful checking: pretesting ensures that acceptance test execution will not reveal any new information. This is demonstration, and nothing more. It has consequences:
- Execution is often the first opportunity that acceptance testers have to get their hands on the software. With little or no opportunity to interact with the software in advance, just how insightful will their preplanned tests be?
- Bugs don’t neatly line up along the course charted by tests. Nor do they conveniently congregate around requirements or other abstractions of system behavior just waiting to be found. Confirmatory requirements based testing will miss all manner of problems.
- Pretesting creates safe pathways through the software. If acceptance testing is confined to these tests it can result in an acceptance decision regardless of the hazards that may lurk beyond these paths.
- Acceptance testers, be they customers, users or their representatives have the potential to bring important insights to testing. They have a different perspective, one that is often centered on the value that the software could bring. This opportunity is wasted if they are made to follow a process that blinds them.
Acceptance testing is often differentiated from other forms of testing in terms of its purpose: whilst earlier tests are focused on finding problems, acceptance testing is sometimes positioned as a confidence building exercise. The risk is that acceptance testing becomes a confidence trick.
The good news is that this risk can be mitigated, even whilst checking many of the boxes that will satisfy project management. A couple of years ago I found myself in an unusual position; working for a vendor yet managing all testing, including acceptance by the customer. This presented a potential conflict of interest that I was determined to avoid. The contract was fixed price and payment was tied to a specific delivery date so the project manager wanted to adopt practices similar to those described above. Fortunately, he also accepted that doing so risked imposing constraints on the quality of acceptance and was willing to entertain alternatives. We agreed on the following:
- Domain and application experts would provide training to the acceptance team prior to testing, and would be on hand to provide coaching throughout.
- User Acceptance Demonstrations would serve to provide basic verification of requirements.
- This would be supplemented by exploratory testing, which would allow the acceptance testers to kick the tires and bring their own perspectives to bear in a more meaningful way that the scripts alone would allow.
- A transparent and visibly fair triage process would be implemented, which would allow the customer to put forth their own prioritization of bugs whilst allowing the project management to intervene should bugs be reported that were beyond the scope of the project.
Project management had the control they needed over scope. The customer was able to get a good feel for the software and the value it would provide. We were able to identify a number of important bugs that would otherwise have escaped us and become warrantee issues. With a little bit of thought, we managed to put the testing back into acceptance testing. Which are you, tester or grifter?