Tag Archives: Scripts

Defaults: Three Tales

A third decade, and some things just don’t seem to change. The following stories have been lightly fictionalised. The names have been changed…

One: The Nineties

Pete didn’t have a clue what he was doing.

He’d made the jump about eighteen months earlier, using his masters to move to a consultancy. Things had looked bright, but after its initial promise, the Next-Big-Thing had turned out to be Nothing-Much-At-All: the market now looked like a coal mine après-Thatcher. So much for his big break in consulting.

He had a young and growing family, had bills to pay, but was determined to keep a mortgage over their heads. He’d been terrified about being laid off and having narrowly dodged the bullet at the last annual RIF he figured his best strategy was making himself useful. The bench was not a healthy place to be. He’d fired up the internal jobs board and scanned the list of open assignments, desperately searching for something, anything, that he might be able to do.

Test Manager, he’d thought, how hard can that be?

The first couple of days hadn’t been too bad. He’d been introduced to some people and sat in a bunch of meetings, eyes glazing over as the team debated the finer points of implementing Soothsayer Enterprise.

It wasn’t until the third day that things got real. “So Pete,” the PM had said, “what about the testing?”

“Urghh”, he’d replied, grasping unconvincingly. THINK, he’d thought, don’t look an idiot in front of the client. What would testing this involve?

“Scripts”, he’d finally blurted out. You need scripts for testing, right? I’m sure I heard that somewhere. “We’ll need scripts – so who’s going to write them, who’s going to approve them, and who’s going to do them?”.

After that, things had settled down. He turned his hand to something he excelled at: spreadsheets. He had spreadsheets of scripts, spreadsheets to track the plans to perform the scripts, spreadsheets to report on the scripts. He was in his element.

Unfortunately, Pete didn’t have a clue what he was doing. Worse: he didn’t know it yet.

Two: The Aughts

Angela was having a bad day, probably the worst since she had joined Delanorme. Things had started out well: first class honours, snapped up by one of the Big Four, aced her GlobalMethodsTM Boot Camp in record time and parachuted into a high-profile presales gig with a leading bank.

Today though, it’ll all gone wrong. Worst of all, she couldn’t figure out why. She’d just heard the news: her bid had been beaten out by TLA Consulting. TLA Consulting! With pretty much no name in the testing business, and no recognized methodology to speak of, TLA had sent some old guy in to counterbid at the last minute. This guy, must be in his late thirties at least, didn’t seem to know anything about testing at all. He’d refused point blank to talk about the number of test scripts needed, rolled his eyes during a discussion about how many test phases would be involved, and seemed to spend most of his time talking with the client about what they were trying to do and what problems they’d had on the last release. Finally, he’d rocked up with a bid twice as high as Angela’s and talked some nonsense about “high volume mechanized checking supplemented by exploration”. Angela had been convinced she had this in the bag.

She checked the time. Twenty minutes: less than half an hour before one of the partners showed up wanting to post mortem. She desperately needed to check her facts, make sure she hadn’t screwed up. Angela fired up her screen, double-clicking on the TestQuoter icon.

Right she thought, Data Testing tab, let’s see…

OK, number of tables, check.

Average number of attributes…let me see, that looks OK.

Complexity factor, medium, seems reasonable.

Expected number of iterations; default. Was that the problem? No, she thought. She remembered her training: always leave that as the default three. No one ever took it seriously if you quoted fewer, or more, than three test cycles.

Her eyes scanned down the screen, her heart beating in her ears as the scrolled to the results box.

“Phew”, she muttered. I’m going to be OK. No matter what went wrong, it wasn’t me, I got the estimate right.

The results box read: 2000 test cases = $1 Million.

Three: The Teens

Watson sat in Zhang’s office, watching skyscrapers slowly emerge from London’s tungsten mists.

“Morning”, said Zhang, walking in, dropping her backpack on a spare seat.

“Hey”. No need to be formal: Watson and Zhang went way back. Whilst Zhang had moved on and up, riding her reputation as someone who could get things done, Watson was sticking it out at the old gig: he’d mentally committed to getting his current project over the line before even considering his options.

“I need your help”, Zhang continued, “your BU’s lit up red like a De Wallen window. Needs fixing fast or Krige’s going to get involved.”

Krige: Avasarala’s fixer-in-chief. Watson knew Zhang well enough to know that this wasn’t idle name dropping or an appeal to authority, if Krige was taking an interest they’d need a solve ASAP. “Go on?”

“It’s the Global Reg Programme: no-one upstairs has a clue what’s going on. And now the Test Governance Group are calling you out as red. Say you haven’t submitted any test reports yet. What’s up?”

Test reports. It didn’t seem to make any difference how many advances were made at the shop floor; this always came up on the big programmes. There was always an information vacuum at the top of the house, always a need to know where the flash points were so the top execs could muster support where it was needed. It was a legitimate need, but it always went the same way: by the time the need had been expressed through multiple layers of organization, it always degenerated into a data collection exercise based on the same assumptions: discrete test phases, script based testing, the same old meaningless metrics. Maybe it was the army of consultants, armed with their methods du jour, drafted in to prop up what was, in effect, a transitory endeavour. Maybe it was the inevitable consequence of miscommunication in how these asks percolated, like a game of Chinese Whispers, through the hierarchy. Maybe these things were just the default assumptions of people who’d never tested anything more than Watson’s patience.

Watson shrugged, mind already working on who would be the appropriate adult. “You know I’m not just going to fall in line for the sake of it, right? If I’m going to fix this, then I’m going to fix it. Do something meaningful.”

Zhang smiled. “I wouldn’t expect anything less”.

“I’ll get on it. Expect more noise before it gets better.”

Devil in the Detail IV


A continued discussion about the arguments for detailed test scripts…

Argument 4: “We need scripts so that we can automate someday”.

Not so very long ago, before the transit strike in Halifax, I used to take the bus to work. I knew that I’d be buying a car at some point, but was looking for the right deal. Perhaps, knowing that I’d soon be purchasing a car, I should have bought myself some wheels, and paid for them out of my transit budget. Maybe even a spare and some winter tires. Of course, I didn’t need wheels to ride the bus, but I could have pretended that this was a cost of using transit so as to make the car look cheaper…

Testing does not require detailed step by step instructions. Automation does. Why hide automation’s true cost? Why? Because often testers and managers want to automate. It is seen as a universal good, or as a way to develop markatable skills. Full disclosure as to its costs might derail the proposed automation initiative.

And in many cases, rightly so, for the argument above mistakenly assumes that automated tests are equivalent to those conducted by humans. Not so. Even some poor soul who has been enslaved to scripted checks and daily test case execution targets has some chance (remote though it might sound) of doing something slightly different, noticing something a little bit odd. Automation does not.

Yet automation stands to be a powerful extension to what testers can accomplish. Such automation cannot be achieved by simply translating human actions into automated ones: rather, it is achieved through understanding what types of actions a machine is better at than us. Now, why would you think that a test script would tell you that?

Devil in the Detail III


A continued discussion about the arguments for detailed test scripts…

Argument 3: “We need scripts so that anyone can test”.

Did you know that anyone can win a medal in the Olympic 100m? Here’s how:

  1. Wait for starting pistol to fire.
  2. Put one foot in front of the other, and repeat really fast.
  3. Collect medal.

Easy! Well of course it isn’t that easy. I suspect that were I to follow that script I’d fall on my face, pull a muscle, or – just maybe – come in last, wheezing.

So how might a tester fall on their face when following a script? What if the software does what the script says, is that a “pass”? Software can misbehave in a near infinite number of ways. It is impossible to encode checks for every instance of system behavior that might be considered a problem. Even were that possible, our stakeholders would balk at the costs of developing, executing and maintaining such checks. Yet a tester following a script is focused on those aspects of behavior that are encoded as checks. They are not primed to observe anything else, nor are they given the means to evaluate any other behaviors. Should anything else occur, they are likely to miss it, else not know what to do with it.

What if the software doesn’t do what the script says, is that a problem? It might be. It might suggest that the person who encoded the check might consider it a problem. Or that it is a legitimate aspect of system behavior not accounted for in the setup of the check. Or that some other condition not encoded in the script has been satisfied. Or that the tester has interpreted the script in a manner other than which it was intended. Again, the tester who relies on a script does not have the means to evaluate any of this. For some further thoughts on this topic, see Doug Hoffman thoughts as to why tests don’t really pass or fail.

There is a significant gap between following a script and the ability to perform such that the desired results of a script are achieved. In the example of the 100m sprint that gap is obvious: it captures only the race itself whilst completely ignoring training and diet. Like sprinters, testers need to train. We need to train our observation skills, our thinking skills, our communication and relationship skills, our information acquisition and synthesis skills. The skills that can help us are many and varied. Also like sprinters, we need the right diet, but I’m not talking about pizza and pop delivered to the lab at silly o’clock at night. The diet testers need is a steady diet of information, and as in the world of nutrition we need to digest the right stuff. Now, some people believe that scripts provide a means for testers to learn how to test a particular solution. Quite apart from scripts being a lousy way to learn, this belief demonstrates a seriously blinkered view as to the kinds of information a tester needs. Scripts might tell you what buttons to press and in which order, they might give you some clues (with the aid of some inference) as to what results to expect when you press those buttons, but they give you no indication of why those results might be important, or what the consequences are of a different result. Nor do scripts provide any insight into any of the human aspects of the project. Scripts are the equivalent of junk food.

So why is it important that “anyone can test”? Sometimes this reflects a desire to mitigate the risks of critical people leaving the project. This is an entirely reasonable concern for which scripts – though an easy answer – do nothing to address. In other cases, this is born of the desire to reduce the costs of testing: if we can get scripts written perhaps we can “juniorize” the team or outsource the testing. This is a case of commoditization. Recently it was suggested to me that projects should always seek to use the lowest cost testers, because testing is “just an overhead anyway”. If you really believe that, please put your money where your mouth is and don’t test your software. While you are at it, I suggest that you update your resume.

User Acceptance Tricks?

Some time ago, I was an SAP consultant. Between projects I configured a variety of demonstration systems for use in presales. These were pretty rough and ready, but they did the job. The trick (and I mean that literally) was to carefully define detailed step by step scripts, test them, and make sure that the demonstrator followed them to the letter. These provided safe pathways; routes through the application that were free of problems. A demonstrator would stray from the path at their peril; the demo would quickly fall apart if they did.

This is analogous to some User Acceptance Testing practices that I’ve observed. Do you recognize this?

The blinkered scripting scam: Acceptance tests will be scripted in advance. They will be traced to requirements. The scripts will be reviewed and pretested by the project. If all the tests pass when executed by the acceptance team, then the software will be accepted.

From a project management perspective this would seem to make sense:

  • It gives the project an opportunity to check that the acceptance team is only testing behavior that is considered to be within the projects scope.
  • It gives the project an opportunity to make sure that acceptance tests will pass before they are formally executed.
  • It helps to ensure that the acceptance team can begin execution just as soon as the software is ready for them.

This is not testing, nor is it even meaningful checking: pretesting ensures that acceptance test execution will not reveal any new information. This is demonstration, and nothing more. It has consequences:

  • Execution is often the first opportunity that acceptance testers have to get their hands on the software. With little or no opportunity to interact with the software in advance, just how insightful will their preplanned tests be?
  • Bugs don’t neatly line up along the course charted by tests. Nor do they conveniently congregate around requirements or other abstractions of system behavior just waiting to be found. Confirmatory requirements based testing will miss all manner of problems.
  • Pretesting creates safe pathways through the software. If acceptance testing is confined to these tests it can result in an acceptance decision regardless of the hazards that may lurk beyond these paths.
  • Acceptance testers, be they customers, users or their representatives have the potential to bring important insights to testing. They have a different perspective, one that is often centered on the value that the software could bring. This opportunity is wasted if they are made to follow a process that blinds them.

Acceptance testing is often differentiated from other forms of testing in terms of its purpose: whilst earlier tests are focused on finding problems, acceptance testing is sometimes positioned as a confidence building exercise. The risk is that acceptance testing becomes a confidence trick.

The good news is that this risk can be mitigated, even whilst checking many of the boxes that will satisfy project management. A couple of years ago I found myself in an unusual position; working for a vendor yet managing all testing, including acceptance by the customer. This presented a potential conflict of interest that I was determined to avoid. The contract was fixed price and payment was tied to a specific delivery date so the project manager wanted to adopt practices similar to those described above. Fortunately, he also accepted that doing so risked imposing constraints on the quality of acceptance and was willing to entertain alternatives. We agreed on the following:

  • Domain and application experts would provide training to the acceptance team prior to testing, and would be on hand to provide coaching throughout.
  • User Acceptance Demonstrations would serve to provide basic verification of requirements.
  • This would be supplemented by exploratory testing, which would allow the acceptance testers to kick the tires and bring their own perspectives to bear in a more meaningful way that the scripts alone would allow.
  • A transparent and visibly fair triage process would be implemented, which would allow the customer to put forth their own prioritization of bugs whilst allowing the project management to intervene should bugs be reported that were beyond the scope of the project.

Project management had the control they needed over scope. The customer was able to get a good feel for the software and the value it would provide. We were able to identify a number of important bugs that would otherwise have escaped us and become warrantee issues. With a little bit of thought, we managed to put the testing back into acceptance testing. Which are you, tester or grifter?

Devil in the Detail II


Continuing with the theme of review and approval as arguments for detailed test scripts…

Argument 2: “We need scripts so that they can be approved prior to execution”.

Like the previous argument, this expresses both a desire and an assumption:

  • DESIRE: We want tests to be approved before they are executed.
  • ASSUMPTION: We can only do that if there are scripts.

Again, we should question if the assumption is valid. It’s much easier for an approver to work through a high level outline of tests than a thousand pages of scripts. I’ve seen approvers asked to do the latter on a number of occasions: I’ve seldom seen an approval result from it.

We should also question the value of such approval. How valid is the approval of tests before they are executed? The tests will change, they must change otherwise many of the insights gained through testing go to waste. As we test, we learn about the software and how it will be used; we learn things that were never considered when the requirements were formulated; we learn implications of the design that were not – could not – be foreseen prior to it being materialized. Further, our learning and observations cause change: as we identify issues with the requirements and design, we drive change in the purpose and implementation of the software. If the tests aren’t adapted to match the evolution of the software, how useful will the tests be? Continuing to follow a plan once it has been established that the plan no longer reflects the situation on the ground is an exercise in futility. The tests must evolve with the software and with our understanding of it. And does every such change require another approval? This creates change inertia, which slows down testing, lengthens the feedback loop and drags down the value of testing.

Where does this desire originate?  In many cases, though demonstrating a lack of understanding as to the nature of discovery, it can be perfectly innocent. However, it can also be a warning signal for something far more sinister:

  • Perhaps the stakeholders of testing have little confidence in the testers: “We want to approve your tests because we don’t believe you are capable of making the right decisions as to how or what to test”. Now, it may be that in such situations the stakeholders have a crystal ball and can predict which tests are required to find all the important bugs, but it is rather more likely that this is evidence of perceived commoditization and the death of trust. If you don’t trust your testers, then replace them – or do a better job of articulating what’s important to you.
  • Perhaps the testers themselves are seeking approval as a means to protect themselves from a witch hunt: “We fear that you will haul us over the coals if we miss a bug – so we want you to approve our tests”. Developers can make bugs but woe betide the tester who misses one? It’s no secret that it impossible to test everything: missing bugs is inevitable. If this is typical of your work environment, take it as indicative of the state of relationships in your workplace. Put some serious thought into whether this can be fixed, or whether you should be considering other options.

Finally it’s worth mentioning that in some cases (for example outsourced testing), both scripts and approvals are a contractual requirement. In these cases it’s worth running through the contract to determine exactly what’s required, and giving thought to how to prevent this from dumbing down the testing.  And have a conversation with however writes the contracts!

More to follow…

Devil in the Detail I


As a test service provider, the most frequent arguments that I hear in favour of detailed scripts concern review and approval. I’ll start with those.

Argument 1: “You need scripts so that we can review what you intend to test”.

This argument expresses two things: a desire and an assumption:

  • DESIRE: We want to understand what you plan to test.
  • ASSUMPTION: We can only do that if there are scripts.

As far as the desire goes, I love it when my stakeholders take an interest in testing. This is a GOOD thing. I want their input and I’m all for review. The more stakeholders that I can get involved in discussing testing the better:  I want to know how they intend to use the software and what keeps them up at night.

Often, this desire stems from a need to simply understand what the testers will be doing, or from a need to gain confidence that the testers are looking at the right things. Yet sometimes there can be a dangerous side to this desire. It is human nature to crave certainty in an uncertain world, and this can manifest as unrealistic expectations as to what can be planned in advance. I’m sure you’ve seen this: huge and elaborate planning efforts that deliver thousand line plans that are broken the moment the project starts. The reality of testing is that we discover and uncover as we go. To ignore what you learn and the new tests that this might suggest is to introduce unnecessary constraints on your testing. There’s value in planning, but none in attempting to hold to a plan when it is clear that it does not reflect the facts on the ground. If you encounter this expectation, and your mission involves more than simply performing a set of predefined checks, then you need to reset this expectation.

Now, to the assumption; that scripts are necessary for reviews. I’d argue the opposite; that detailed scripts are the death of a review. If you hit your reviewer with a thousand page dump of step by step test scripts from your test management tool (I call this a QC barf), expect narcolepsy rather than any meaningful input. A simple list of test cases is much more digestible and conducive to engaging stakeholders in a useful conversation. A mind map that captures test ideas (see Christin Wiedemann or Darren McMillan’s blog) might be even better when it comes to finding gaps and thinking up new ideas. Either of these approaches comes with a significantly lower price tag, and far greater flexibility, than detailed scripts.

More to follow…


Devil in the Detail: Prologue

On my first real testing job I hadn’t a clue what I was doing, and no one around me had any testing experience either. This was when the web was in its infancy (I was still browsing with Lynx and a 2400 baud modem) and it didn’t even really occur to me that there might be books on the subject. 

I did the only sensible thing: I made it up as I went along. I can only assume that I’d subconsciously overheard people talking about testing at some point, because I did know one thing: testing needed scripts. I dutifully set about talking to the team, reviewing the design and writing a set of step by step scripts that covered the requirements. The results were not exactly outstanding, but I muddled through.

Over the next few years, I relied on scripts less: I found that they slowed me down, that I could test more productively without them. Much of the time I’d define lists of tests, or in some cases matrices describing combinations of data. These served me well. I was documenting to a level that someone with knowledge of the software could execute. Whenever I had to hand over test cases, I’d add some supplementary information and sit down and train the new testers on using the software.

On one project, one of my team told me “I prefer exploratory testing to scripts”. I wasn’t really sure what she meant. It wouldn’t be long before I found out: this was shortly after a deeply embarrassing interview had convinced me to start learning more about testing, and I was spending every spare moment reading anything I could find. 

My reaction when I first read about exploratory testing? I couldn’t understand what the fuss was about. This wasn’t anything different or special, this was just testing! Every time I’d ever played with an application to figure out how to test it, I’d been doing ET. Every time I’d prodded and poked at a program to figure out why it wasn’t quite doing what I’d expected, I’d been doing ET. Every time I’d noticed something new and tried out new tests, I’d been doing ET. I found it almost inconceivable that anyone in testing didn’t do this.

As I started managing larger testing teams, I grew to appreciate ET more. I found a surprising degree of script dependence and inflexibility amongst many testers. Encouraging the use of exploratory testing helped to correct that. 

What constantly surprises me is the range of arguments that I came across in favour of detailed “idiot scripts”, scripts that are detailed enough that any idiot could use them. This series of posts will take a look at some of those arguments.