managing the testing process, 3rd edition

Managing the TestingProcess Practical Tools and Techniques for Managing Software and Hardware Testing Third Edition Rex Black Wiley Publishing, Inc... In the course of learning how to ma

Managing the Testing

Process

Practical Tools and Techniques

for Managing Software and

Hardware Testing

Third Edition Rex Black

Wiley Publishing, Inc.

Copyright © 2009 by Rex Black All rights reserved.

Published by Wiley Publishing, Inc., Indianapolis, Indiana

Published simultaneously in Canada

ISBN: 978-0-470-40415-7

Manufactured in the United States of America

10 9 8 7 6 5 4 3 2 1

No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form

or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as permitted under Sections 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee

to the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 646-8600 Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, or online at http://www.wiley.com/go/permissions.

Limit of Liability/Disclaimer of Warranty:The publisher and the author make no representations or warranties with respect to the accuracy or completeness of the contents of this work and specifically disclaim all warranties, including without limitation warranties of fitness for a particular purpose No warranty may be created or extended by sales or promotional materials The advice and strategies contained herein may not be suitable for every situation This work is sold with the understanding that the publisher is not engaged in rendering legal, accounting, or other professional services If professional assistance is required, the services of a competent professional person should be sought Neither the publisher nor the author shall be liable for damages arising herefrom The fact that an organization or Web site is referred to in this work as a citation and/or a potential source of further information does not mean that the author or the publisher endorses the information the organization

or Web site may provide or recommendations it may make Further, readers should be aware that Internet Web sites listed in this work may have changed or disappeared between when this work was written and when it is read.

For general information on our other products and services please contact our Customer Care Department within the United States at (877) 762-2974, outside the United States at (317) 572-3993 or fax (317) 572-4002.

Wiley also publishes its books in a variety of electronic formats Some content that appears in print may not be available in electronic books.

Library of Congress Control Number:2009929457

Trademarks:Wiley and the Wiley logo are trademarks or registered trademarks of John Wiley & Sons, Inc and/or its affiliates, in the United States and other countries, and may not be used without written permission All other trademarks are the property of their respective owners Wiley Publishing, Inc is not associated with any product or vendor mentioned in this book.

With a quarter-century of software and systems engineering experience,

Rex Black is President of RBCS (www.rbcs-us.com), a leader in software,hardware, and systems testing For more than a dozen years, RBCS hasdelivered services in consulting, outsourcing, and training for software andhardware testing Employing the industry’s most experienced and recognizedconsultants, RBCS conducts product testing, builds and improves testinggroups, and hires testing staff for hundreds of clients worldwide Rang-ing from Fortune 20 companies to start-ups, RBCS clients save time andmoney through improved product development, decreased tech support calls,improved corporate reputation, and more

As the leader of RBCS, Rex is the most prolific author practicing in the

field of software testing today His popular first book, Managing the Testing

Process, now in its third edition, has sold more than 30,000 copies around the

world, including Japanese, Chinese, and Indian releases His five other books

on testing, Critical Testing Processes, Foundations of Software Testing, Pragmatic

Software Testing, Advanced Software Testing: Volume I, and Advanced Software Testing: Volume II, have also sold tens of thousands of copies, including Hebrew,

Indian, Chinese, Japanese, and Russian editions He has contributed to 10 otherbooks as well He has written more than 25 articles, presented hundreds ofpapers, workshops, and seminars, and given about 30 keynote speeches atconferences and events around the world Rex is a former president of boththe International Software Testing Qualifications Board and the AmericanSoftware Testing Qualifications Board

When he is not working with clients around the world, developing orpresenting a training seminar, or in his office, Rex spends time at home

or around the world with his wife and business partner, Laurel Becker; hisdaughters Emma Grace and Charlotte Catherine; and his faithful canine friendsHank and Cosmo

iii

This book is a third edition, and that happens only when the first edition andsecond edition are successful So, first off, I’d like to thank those of you whobought the first edition or second edition I hope it’s proven a useful referencefor you A special thanks goes to RBCS clients who used these books on theirprojects, attendees who provided evaluations of RBCS training courses, andreaders who wrote reviews or sent me emails about the book I have addressedyour suggestions for improvement in this third edition.

A book gets into people’s hands only through a lot of behind-the-sceneshard work by a publisher’s team A special thanks to the fine people at Wileywho helped bring this book to fruition, especially Kelly Talbot and RobertElliott RBCS associate Judy McKay provided valuable technical reviewing

help I’d also like to thank Ben Ryan, who shepherded Managing the Testing

Process along through the first two editions, starting in 1998 I’d also like to

thank my friends at Microsoft Press who helped me with the first edition: ErinO’Connor, John Pierce, Mary Renaud, Ben Ryan (again), and Wendy Zucker

In the course of learning how to manage test projects, I have worked withmany talented professionals as a tester, test manager, and consultant The list

of people who helped me is literally too long to include here, but my gratitude

to all of my colleagues and clients is immense

The material in this book appears in one-day, two-day, and three-day testmanagement courses that RBCS associates and I have presented all around theworld I thank all the attendees of those seminars for their help making thismaterial better in the third edition

Of course, my appreciation goes out to all my past and current colleagues,subcontractors, employees, clients, and employers I especially want to thankthe clients who graciously agreed to the use of data and documentation fromtheir projects to illustrate many of the tools and techniques I discuss

v

Four people I want to name specifically in this regard are Judy McKay,Andrew Brooks, Jodi Mullins, and Steven Gaudreau Judy is a director ofquality assurance at large network equipment company Andrew Brooks isvice president, CA Network and Voice Management Quality Assurance JodiMullins is senior software engineer, CA Network and Voice Management TestAutomation Steven Gaudreau is software engineer, CA Network and VoiceManagement Test Automation Each shared specific case studies, authored bythem, about topics central to a chapter of each book I really appreciate theirvaluable, practitioner insights.

Please attribute all errors, omissions, mistakes, opinions, and bad jokes inthis book solely to me

In the realm of ‘‘without whom,’’ of course, I thank my parents, Rex, Sr andCarolynn, for their love and support over the years My greatest appreciation

goes to my wife and business partner, Laurel Becker Managing the Testing

Process has taken me away from a lot of things in my life, three times now, but

I especially appreciate my wife’s support in terms of her own time given upfor me

I’ve changed a few of my ideas since I wrote the first and second editions,but the biggest changes in my life have involved the arrival of my daughters.Along with having a burst of wisdom that led me to marry Laurel, I have to saythat Emma Grace and Charlotte Catherine are the greatest things to happen

in my life All parents have dreams for their children’s success, and I hope that

my two beautiful and inspirational daughters have the same luck and success

in their careers that I have had Whatever Emma and Charlotte choose to do,this book is dedicated to them, with the utmost of a father’s love

Introduction xxiii Chapter 1 Defining What’s on Your Plate: The Foundation of a Test

Chapter 2 Plotting and Presenting Your Course: The Test Plan 49 Chapter 3 Test-System Architecture, Cases, and Coverage 79 Chapter 4 An Exciting Career in Entomology Awaits You: A

Chapter 5 Managing Test Cases: The Test-Tracking Spreadsheet 199 Chapter 6 Tips and Tools for Crunch Mode: Managing the Dynamic 257 Chapter 7 Stocking and Managing a Test Lab 293 Chapter 8 Staffing and Managing a Test Team 319 Chapter 9 The Triumph of Politics: Organizational Challenges for

Chapter 10 Involving Other Players: Distributed Testing, outsourcing,

Chapter 11 Economics of Testing: Fiscal Context 475 Chapter 12 Testing Implications of Project and Process: Situational

vii

Appendix A Hardware-Testing Fundamentals: An Introduction for

Software-Testing Professionals 553 Appendix B Omninet: The Internet Everywhere Marketing

Introduction xxiii

Chapter 1 Defining What’s on Your Plate: The Foundation of a Test

ix

Identify and Assess: Process Options for Quality-Risk

Chapter 2 Plotting and Presenting Your Course: The Test Plan 49

Clarity, Pertinence, and Action 76

Chapter 3 Test System Architecture, Cases, and Coverage 79

It’s Not Saint Paul’s, But .Principles for Test System

Miscellaneous Best Practices and Principles for Quality Test

Avoiding the Dreaded Test Escape: Coverage and

What if I Can’t Repeat All the Tests? Alternative

‘‘There’s a Lesson to Be Learned Here .’’: Test Case

Bonus Case Study 138

Putting the Tracking in Bug Tracking: Adding Dynamic

What the Bug Relates To: Subsystem, Configuration, and

How Long Was the Bug Around? Close Date and the

Extracting Metrics from the Bug-Tracking Database 179

Chapter 5 Managing Test Cases: The Test-Tracking Spreadsheet 199

Are We Getting as Much Work Done as Planned? Charting

Are We Testing What We Promised? Charting Test and Bug

Test Status in a Nutshell: Building a Balanced Scorecard or

Chapter 6 Tips and Tools for Crunch Mode: Managing the Dynamic 257

Moving Forward While Getting All the Facts: The Desire for

Dependencies, Schedules, and Reminders: The Importance of

It Won’t Install Itself, Either: Configuring the Test

‘‘The Hobgoblin of Small Minds’’ Is Your Friend: Auditing

Test-Result Misinterpretation: Minimizing False Positives

‘‘I Wish You a Merry Dragon-Boat Festival .’’: When Crunch

A Spider’s Web of Connections: Managing Test

The Pieces and How They Connect: An Entity-Relationship

The Assets, How You Use Them, and Where They Live:

Chapter 7 Stocking and Managing a Test Lab 293

Chapter 8 Staffing and Managing a Test Team 319

The Right Person for the Job: What Kind of People Make

Education, Training, and Certification 331

Extending Your Talent: Using Temporary Experts and

Working with Other Managers: Directions of Test

Bringing Management to Your Reality: Communicating

‘‘How about a Third Shift, and Weekends, and .’’: The

Your Partners in Building Quality Systems: Development

Help Desk, Customer Support, or Technical Support:

Testing in the Dark: Should You Proceed without

Presenting the Results: The Right Message, Delivered

‘‘You Can Tell the Pioneers ’’: The Effect of Early Adoption

Chapter 10 Involving Other Players: Distributed Testing, outsourcing,

Case Study 2 461Key Differences between Testing Service Providers and

Chapter 11 Economics of Testing: Fiscal Context 475

Is Quality Free? The Economic Justification for the Testing

Test-Manager Budgeting Faux Pas: Obstacles the Test

Regrettable Necessity: Obstacles the Testing Reality

Communication Breakdowns: Management Blind Spots

Chapter 12 Testing Implications of Project and Process: Situational

System, Subsystem, Commercial Off-the-Shelf Software, and

Process Brake Pedals 521

Receiving Code after Inconsistent (and Often Inadequate)

Appendix A Hardware Testing Fundamentals: An Introduction for

Software Testing Professionals 553

Required release date 570

Kiosk OS/Browser/Connection Speed Configuration

Appendix D Bibliography, Related Readings, and Other Resources 591

Project Test Services 598

So, you are responsible for managing a computer hardware or software testproject? Congratulations! Maybe you’ve just moved up from test engineering

or moved over from another part of the development team, or maybe you’vebeen doing test projects for a while Whether you are a test manager, a develop-ment manager, a technical or project leader, or an individual contributor withsome level of responsibility for your organization’s test and quality assuranceprogram, you’re probably looking for some ideas on how to manage theunique beast that is a test project

This book can help you The first edition, published in 1999, and thesecond edition, published in 2002, have sold over 35,000 copies in the lastdecade There are popular Indian, Chinese, and Japanese editions, too Clients,colleagues, readers, training attendees, and others have read the book, writingreviews and sometimes sending helpful emails, giving me ideas on how to im-prove and expand it So, thanks to all of you who read the first and secondeditions, and especially to those who have given me ideas on how to makethis third edition even better

This book contains what I wish I had known when I moved from ming and system administration to test management It shows you how todevelop some essential tools and apply them to your test project It offerstechniques that can help you get and use the resources you need to succeed Ifyou master the basic tools, apply the techniques to manage your resources, andgive each area just the right amount of attention, you can survive managing atest project You’ll probably even do a good job, which might make you a testproject manager for life, like me

program-xxiii

The Focus of This Book

I’ve written Managing the Testing Process for several reasons First, many

projects suffer from a gap between expectations and reality when it comes

to delivery dates, budgets, and quality, especially between the individualcontributors creating and testing the software, the senior project managers,and the users and the customers Similarly, computer hardware developmentprojects often miss key schedule and quality milestones Effective testingand clear communication of results as an integrated part of a project riskmanagement strategy can help

Second, when I wrote the first edition, there was a gap in the literature

on software and hardware testing We had books targeting the low-levelissues of how to design and implement test cases, as well as books tellingsophisticated project managers how to move their products to an advancedlevel of quality using concepts and tools such as the Capability MaturityModel, software quality metrics, and so forth However, I believe that testmanagers like us need a book that addresses the basic tools and techniques,the bricks and mortar, of test project management While there are now anumber of books addressing test management, I believe this book remainsunique in terms of its accessibility and immediate applicability to the first-timetest manager while also offering guidance in how to incrementally improve

a foundational test management approach It also offers a proven approachthat works for projects that include substantial hardware development orintegration components

The tips and tools offered in this book will help you plan, build, and execute

a structured test operation As opposed to the all-too-common ad hoc or purelyreactive test project, a structured test operation is planned, repeatable, anddocumented, but preserves creativity and flexibility in all the right places.What you learn here will allow you to develop models for understandingthe meaning of the myriad data points generated by testing so that you caneffectively manage what is often a confusing, chaotic, and change-ridden area

of a software or hardware development project This book also shows youhow to build an effective and efficient test organization

To that end, I’ve chosen to focus on topics unique to test management in thedevelopment and maintenance environments Because they’re well covered inother books, I do not address two related topics:

Basic project management tools such as work-breakdown structures, Gantt charts, status reporting, and people management skills.Asyou move into management, these tools will need to be part of yourrepertoire, so I encourage you to search out project management

books — such as the ones listed in the bibliography in Appendix D — to

help you learn them A number of excellent training courses and

certifications currently exist for project management as well

Computer hardware production testing.If your purview includes this

type of testing, I recommend books by W Edwards Deming, Kaoru

Ishikawa, and J M Juran as excellent resources on statistical quality trol, as well as Patrick O’Connor’s book on reliability engineering; see

con-the bibliography in Appendix D for details on books referenced here

Software production, in the sense of copying unchanging final versions todistribution media, requires no testing However, both hardware and softwareproduction often include minor revisions and maintenance releases You canuse the techniques described in this book to manage the smaller test projectsinvolved in such releases

The differences between testing software and hardware are well mented, which might make it appear, at first glance, that this book is headed

docu-in two directions I have found, however, that the differences between thesetwo areas of testing are less important from the perspective of test projectmanagement than they are from the perspective of test techniques This makessense: hardware tests software, and software tests hardware Thus, you canuse similar techniques to manage test efforts for both hardware and softwaredevelopment projects

Canon or Cookbook?

When I first started working as a test engineer and test project manager, Iwas a testing ignoramus While ignorance is resolvable through education,some of that education is in the school of hard knocks Ignorance can lead

to unawareness that the light you see at the end of the tunnel is actually anoncoming train ‘‘How hard could it be?’’ I thought ‘‘Testing is just a matter

of figuring out what could go wrong, and trying it.’’

As I soon discovered, however, the flaws in that line of reasoning lie in threekey points:

The tasks involved in ‘‘figuring out what could go wrong, and tryingit’’ — that is, in designing good test cases — are quite hard indeed Manyauthors have written good books on test case engineering, particularly

in the last two decades Unfortunately, my university professors didn’tteach about testing, even though Boris Beizer, Bill Hetzel, and GlenfordMyers had all published on the topic prior to or during my college career

As software engineering enters its sixth decade, that has begun to change.However, even at prestigious universities the level of exposure to testingthat most software-engineers-in-the-making receive remains too low

Testing does not go on in a vacuum Rather, it is part of an overallproject — and thus testing must respond to real project needs, not to thewhims of hackers playing around to see what they can break In short,test projects require test project management.

The prevalence of the ‘‘how hard can testing be’’ mindset only serves

to amplify the difficulties that testing professionals face Once we’velearned through painful experience exactly how hard testing can be, itsometimes feels as if we are doomed — like a cross between Sisyphusand Dilbert — to explain, over and over, on project after project, why thistesting stuff takes so long and costs so much money

Implicit in these points are several complicating factors One of the mostimportant is that the capability of an organization’s test processes can varyconsiderably: testing can be part of a repeatable, measured process, or an adhoc afterthought to a chaotic project In addition, the motivating factors — thereasons why management bothers to test — can differ in both focus andintensity Managers motivated by fear of repeating a recent failed projectsee testing differently than managers who want to produce the best possibleproduct, and both motivations differ from those of people who organize testefforts out of obligation but assign them little importance Finally, testing istightly connected to the rest of the project, so the test manager is often subject

to a variety of outside influences These influences are not always benign whenscope and schedule changes ripple through the project

These factors make it difficult to develop a how to guide for planning and

executing a test project As academics might say, test project management doesnot lend itself to the easy development of a canon ‘‘Understand the followingideas and you can understand this field’’ is a difficult statement to apply totest management And the development of a testing canon is certainly not anundertaking I’ll tackle in this book

Do you need a canon to manage test projects properly? I think not Instead,consider this analogy: I am a competent and versatile cook, an amateur chef

I will never appear in the ranks of world-renowned chefs, but I regularlyserve passable dinners to my family I have successfully prepared a number

of multicourse Thanksgiving dinners, some in motel kitchenettes I masteredproducing an edible meal for a reasonable cost as a necessity while working

my way through college In doing so, I learned how to read recipes out of acookbook, apply them to my immediate needs, juggle a few ingredients hereand there, handle the timing issues that separate dinner from a sequence ofsnacks, and play it by ear

An edible meal at a reasonable cost is a good analogy for what yourmanagement wants from your testing organization This book, then, can serve

as a test project manager’s cookbook, describing the basic tools you need andhelping you assemble and blend the proper ingredients

The Tools You Need

Several basic tools underlie my approach to test management:

A solid quality risk analysis.You can’t test everything Therefore, a key

challenge to test management is deciding what to test You need to

find the important bugs early in the project Therefore, a key challenge

to test management is sequencing your tests You sometimes need to

drop tests due to schedule pressure Therefore, a key challenge to test

management is test triage in a way that still contains the important

risks to system quality You need to report test results in terms that are

meaningful to non-testers Therefore, a key challenge to test

manage-ment is tracking and reporting residual levels of risk as test execution

continues Risk based testing, described in this book, will help you

do that

A thorough test plan.A detailed test plan is a crystal ball, allowing

you to foresee and prevent potential crises Such a plan addresses

the issues of scope, quality risk management, test strategy, staffing,

resources, hardware logistics, configuration management, scheduling,

phases, major milestones and phase transitions, and budgeting

A well-engineered system.Good test systems ferret out, with wicked

effectiveness, the bugs that can hurt the product in the market or reduce

its acceptance by in-house users Good test systems mitigate risks to

system quality Good test systems build confidence when the tests

finally pass and the bugs get resolved Good test systems also produce

credible, useful, timely information Good test systems possess internal

and external consistency, are easy to learn and use, and build on a set

of well-behaved and compatible tools I use the phrase good test system

architecture to characterize such a system The word architecture fosters a

global, structured outlook on test development within the test team It

also conveys to management that creating a good test system involves

developing an artifact of elegant construction, with a certain degree of

permanence

A state-based bug tracking database.In the course of testing, you and

your intrepid test team will find lots of bugs, a.k.a issues, defects, errors,problems, faults, and other less-printable descriptions Trying to keep

all these bugs in your head or in a single document courts immediate

disaster because you won’t be able to communicate effectively within

the test team, with programmers, with other development team peers,

or with the project management team — and thus won’t be able to

contribute to increased product quality You need a way to track each

bug through a series of states on its way to closure I’ll show you how

to set up and use an effective and simple database that accomplishesthis purpose This database can also summarize the bugs in informativecharts that tell management about projected test completion, productstability, system turnaround times, troublesome subsystems, and rootcauses.

A comprehensive test-tracking spreadsheet.In addition to keepingtrack of bugs, you need to follow the status of each test case Does theoperating system crash when you use a particular piece of hardware?Does saving a file in a certain format take too long? Which release of thesoftware or hardware failed an important test? A simple set of work-sheets in a single spreadsheet can track the results of every single testcase, giving you the detail you need to answer these kinds of questions.The detailed worksheets also roll up into summary worksheets thatshow you the big picture What percentage of the test cases passed? Howmany test cases are blocked? How long do the test suites really take

to run?

A simple change management database.How many times have youwondered, ‘‘How did our schedule get so far out of whack?’’ Littlediscrepancies such as slips in hardware or software delivery dates,missing features that block test cases, unavailable test resources, andother seemingly minor changes can hurt When testing runs late, thewhole project slips You can’t prevent test-delaying incidents, butyou can keep track of them, which will allow you to bring delays tothe attention of your management early and explain the problemseffectively This book presents a simple, efficient database that keepsthe crisis of the moment from becoming your next nightmare

A solid business case for testing.What is the amount of money thattesting saves your company? Too few test managers know the answers

to this question However, organizations make tough decisions aboutthe amount of time and effort to invest in any activity based on acost benefit analysis I’ll show you how to analyze the testing return

on investment, based on solid, well established quality managementtechniques

This book shows you how to develop and apply these basic tools to yourtest project, and how to get and use the resources you need to succeed.I’ve implemented them in the ubiquitous PC-based Microsoft Office suite:Excel, Word, Access, and Project You can easily use other office-automationapplications, as I haven’t used any advanced features

The Resources You Need

In keeping with our culinary analogy, you also need certain ingredients, orresources, to successfully produce a dish In this testing cookbook, I showyou how I assemble the resources I need to execute a testing project Theseresources include some or all of the following:

A practical test lab.A good test lab provides people — and

computers — with a comfortable and safe place to work This lab,

far from being Quasimodo’s hideout, needs many ways to communicatewith the development team, the management, and the rest of the

world You must ensure that it’s stocked with sufficient software

and hardware to keep testers working efficiently, and you’ll have to

keep that software and hardware updated to the right release levels

Remembering that it is a test lab, you’ll need to make it easy for

engineers to keep track of key information about system configurations

Test engineers and technicians.You will need a team of

hardwork-ing, qualified people, arranged by projects, by skills, or by a little

of both Finding good test engineers can be harder than finding

good development engineers How do you distinguish the

bud-ding test genius from that one special person who will make your

life as a manager a living nightmare of conflict, crises, and lost

productivity? Sometimes the line between the two is finer than

you might expect And once you have built the team, your work

really begins How do you motivate the team to do a good job?

How do you defuse the land mines that can destroy motivation?

Contractors and consultants.As a test manager, you will probably use

outsiders, hired guns who work by the hour and then disappear when

your project ends I will help you classify the garden-variety high-tech

temporary workers, understand what makes them tick, and resolve the

emotional issues that surround them When do you need a contractor?

What do contractors care about? Should you try to keep the good ones?

How do you recognize those times when you need a consultant?

External test labs, testing services providers, and vendors.In certain

cases, it makes sense to do some of the testing outside the walls of

your own test lab — for instance, when you are forced to handle spikes

or surprises in test workloads You might also save time and money

by leveraging the skills, infrastructure, and equipment offered by

external resources such as testing labs and testing services providers

What can these labs and vendors really do for you? How can youuse them to reduce the size of your test project without creating

dangerous coverage gaps? How do you map their processes and

results onto yours? How does outsourcing fit into your test effort?

Of course, before you can work with any of these resources, you have toassemble them As you might have learned already, management is neverexactly thrilled at the prospect of spending lots of money on equipment to teststuff that — in their view — ‘‘ought to work anyway.’’ With that in mind, I’vealso included some advice about how to get the green light for the resourcesyou really need

The concepts also scale across distributed projects I’ve used the tools

to manage multiple projects simultaneously from a laptop computer inhotel rooms and airport lounges around the world I’ve used these tools

to test market-driven end-user systems and in-house information technologyprojects While context does matter, I’ve found that adaptations of the concepts

in this book apply across a broad range of settings

Simple and effective, the tools incorporate the best ideas from industrystandards such as the IEEE 829 Standard for Software and System Test Docu-mentation and bring you in line with the best test management practices andtools at leading software and hardware vendors I use these tools to organize

my thinking about my projects, to develop effective test plans and test suites,

to execute the plans in dynamic high-technology development environments,and to track, analyze, and present the results to project managers Likewise, mysuggestions on test resource management come from successes and failures atvarious employers and clients

Because context matters, the final two chapters discuss the importance

of fitting the testing process into the overall development or maintenanceprocess This involves addressing issues such as organizational context, theeconomic aspects of and justifications for testing, life cycles and methodologiesfor system development, and increasing test process capability, including testprocess assessment and maturity models

Using This Book

Nothing in this book is based on Scientific Truth, double-blind studies, demic research, or even flashes of brilliance It is merely about what hasworked — and continues to work — for me on the dozens of test projects Ihave managed, what has worked for the clients that my company, RBCS, hasthe good fortune to serve, and what has worked for the thousands of peoplewho have attended RBCS training courses You might choose to apply theseapproaches as is, or you might choose to modify them You might find all oronly some of my approaches useful

aca-Along similar lines, this is not a book on the state of the art in test niques, test theory, or the development process This is a book on testmanagement, both hardware and software, as I have practiced it In terms

tech-of development processes — best practices or your company’s practices — theonly assumption I make is that you as the test manager became involved

in the development project with sufficient lead time to do the necessary testdevelopment Chapter 12 addresses different development processes I haveseen and worked within I cover how the choice of a development life cycleaffects testing

Of course, I can’t talk about test management without talking about testtechniques to some extent Because hardware and software test techniquesdiffer, you might find some of the terms I use unclear or contradictory toyour usage of them I have included a glossary to help you decipher thehardware examples if you’re a software tester, and vice versa Finally, the testmanager is usually both a technical leader and a manager, so make sure youunderstand and use best practices, especially in the way of test techniques, foryour particular type of testing Appendix D includes a listing of books that canhelp you brush up on these topics if needed

This book is drawn from my experiences, good and bad The badexperiences — which I use sparingly — are meant to help you avoid some

of my mistakes I keep the discussion light and anecdotal The theorybehind what I’ve written, where any exists, is available in books listed in thebibliography in Appendix D

I find that I learn best from examples, so I have included lots of them.Because the tools I describe work for both hardware and software testing, Ibase many examples on one of these two hypothetical projects:

Most software examples involve the development of a browser-basedword-processing package named SpeedyWriter, being written by Soft-ware Cafeteria, Inc SpeedyWriter has all the usual capabilities of afull-featured word processor, plus network file locking, Web integration,

and public-key encryption SpeedyWriter includes various add-ins fromother vendors.1

Most hardware examples refer to the development of a server namedDataRocket, under development by Winged Bytes, LLP DataRocket

is intended to serve a powerful, high-end database, application, andWeb server It runs multiple operating systems Along with third-partysoftware, Winged Bytes plans to integrate hardware from vendors aroundthe world

As for the tools discussed in this book, you can find examples of these at

In those chapters that describe the use of these tools, I include information toguide you in the use and study of these templates and case studies should youwant to do so That way, you can use these resources to bootstrap your ownimplementation of the tools These tools are partially shown in figures in thechapters in which I describe them However, screen shots can only tell you somuch Therefore, as you read the various chapters, you might want to openand check out the corresponding case studies and templates from the Web site

to gain a deeper understanding of how the tools work

Please note that the tools supplied with the book are usable, but containonly small amounts of dummy data This data should not be used to deriveany rules of thumb about bug counts, defect density, predominant qualityrisks, or any other metric to be applied to other projects I developed the toolsprimarily to illustrate ideas, so some of the sophisticated automation that youwould expect in a commercial product won’t be there If you intend to usethese tools in your project, allocate sufficient time and effort to adapt andenhance them for your context For large, complex projects, or for situationswhere test management is an ongoing activity, you’ll want to consider buyingcommercial tools

For those wanting to practice with the tools before putting them into use

on a real project, I have included exercises at the end of each chapter Formany of these exercises, you can find solutions at www.rbcs-us.com Theseexercises make this book suitable as the test management textbook for a course

on testing, software engineering, or software project management Given thattesting is increasingly seen by enlightened project managers as a key part ofthe project’s risk management strategy, including material such as this as part

of a college or certification curriculum makes good sense

Finally — in case you haven’t discovered this yet — testing is not a fiefdom

in which one’s cup overfloweth with resources and time I have found that it’scritical to focus on testing what project managers really value Too often in the

might have struck readers as a bizarre concept Well, to those at Google, I say, ‘‘You’re welcome’’ for the idea!

past I’ve ended up wrong-footed by events, spending time handling trivialities

or minutiae while important matters escaped my attention Those experiencestaught me to recognize and attend to the significant few and ignore the trivialmany The tools and techniques presented here can help you do the same,especially the risk-based testing elements A sizeable number of test groupsare disbanded in their first couple of years This book will help keep you out

of that unfortunate club

Although it’s clearly more than simply hanging onto a job, success in testmanagement means different things to different people In my day-to-daywork, I measure the benefits of success by the peace of mind, the reduction instress, and the enhanced professional image that come from actually managingthe testing areas in my purview rather than reacting to the endless sequence

of crises that ensue in ad hoc environments I hope that these tools and ideaswill contribute to your success as a testing professional

What’s New and Changed in the Third Edition?

For those of you who read the second edition and are wondering whether tobuy this third edition, I’ve included the following synopsis of changes andadditions:

I’ve split the final chapter into two detailed chapters on the importance

of fitting the testing process into the overall development or nance process I address organizational context, the economic aspects ofand justifications for testing, life cycles and methodologies for systemdevelopment, test process assessment, and process maturity models

mainte-I have addressed the mainte-IEEE 829-2008 standard, which came out as mainte-I startedwork on this book This new version of the IEEE 829 standard includesnot only document templates, but also discussion on the testing process.While I’m not endorsing the complete adoption of this standard on yourprojects, I believe it does provide useful ideas and food for thought

I also added some new metrics The templates include the tools togenerate those metrics Some of the templates originally published withthe book, while usable, contained minor errors Readers of the first andsecond editions — being test professionals — caught and pointed outthese errors to me I have corrected those mistakes

In addition to case studies, I have added some exercises Some of thesecome from RBCS’s live and e-learning course ‘‘Managing the Testing Pro-cess,’’ some carried over from the second edition, and some are adapted

from Pragmatic Software Testing You can use these exercises for self-study,

as part of a book club, or for classroom education (Some professors have

selected this book as a textbook for a software testing course.) Solutions

to many of these exercises are now available atwww.rbcs-us.com

Finally, little has changed in terms of the challenges facing those whomanage test projects since I wrote the first and second editions Everytime my associates teach RBCS’s ‘‘Managing the Testing Process’’ classes,which are drawn directly from this book, at least one attendee tells

me, ‘‘It’s amazing how every issue we’ve talked about here in class

is something that has come up for me on my projects.’’ However, Ihave learned some new tricks and broadened my mind For example,Agile project methodologies are now quite popular, so I’ve incorporatedmaterial to discuss the challenges that Agile techniques pose for testingand how you can manage these challenges

If you read the second edition, enjoyed it, and found it useful, I think thesechanges and additions will make this third edition even more useful to you

Defining What’s on Your Plate:

The Foundation of a

Test Project

Testing requires a tight focus It’s easy to try to do too much You could run an

infinite number of tests against any nontrivial piece of software or hardware.Even if you try to focus on what you think might be ‘‘good enough’’ quality,you can find that such testing is too expensive or that you have trouble figuringout what ‘‘good enough’’ means for your customers and users Before I start

to develop the test system — the testware, the test environment, and the test

process — and before I hire the test team, I figure out what I might test, then what I should test, and finally what I can test Determining the answers to these

questions helps me plan and focus my test efforts

What I might test are all those untested areas that fall within the purview

of my test organization On every project in which I’ve been involved, someamount of the test effort fell to organizations outside my area of responsibility.Testing an area that another group already covered adds little value, wastestime and money, and can create political problems for you

What I should test are those untested areas that directly affect the customers’and users’ experience of quality People often use buggy software and com-puters and remain satisfied nevertheless Either they never encounter the bugs

or the bugs don’t significantly hinder their work Our test efforts should focus

on finding the critical defects that will limit people’s ability to get work donewith our products

What I can test are those untested, critical areas on which my limitedresources are best spent Can I test everything I should? Not likely, given theschedule and budget I usually have available.1On most projects, I must make

The Art of Software Testing.

1

tough choices, using limited information, on a tight schedule I also need tosell the test project to my managers to get the resources and the time I need.

What You Might Test: The Extended Test Effort

On my favorite software and system projects, testing was pervasive By this,

I mean that a lot of testing went on outside the independent test team

In addition, testing started early This arrangement not only made sensetechnically, but also kept my team’s workload manageable This sectionuses two lenses to examine how groups outside the formal test organization

contribute to testing The first lens is the level of focus — the granularity — of

a test The second is the type of testing performed within various test phases.Perhaps other organizations within your company could be (or are) helpingyou test

From Microscope to Telescope: Test Granularity

Test granularity refers to the fineness or coarseness of a test’s focus A

fine-grained test case allows the tester to check low-level details, often internal

to the system A coarse-grained test case provides the tester with informationabout general system behavior You can think of test granularity as runningalong a spectrum ranging from structural (white-box) to behavioral (black-boxand live) tests, as shown in Figure 1-1

Programmers DB/Net/System Admins Electrical/Mechanical Engineers

Tech Support/Help Desk Sales/Marketing Business Analysts/Users

Live (Alpha, Beta, or Acceptance)

Behavioral (Black-Box)

Structural (White-Box)

Figure 1-1 The test granularity spectrum and owners

Structural (White-Box) Tests

Structural tests (also known as white-box tests and glass-box tests) find bugs in

low-level structural elements such as lines of code, database schemas, chips,

subassemblies, and interfaces The tester bases structural tests on how a system

operates For example, a structural test might reveal that the database thatstores user preferences has space to store an 80-character username, but thatthe field allows the user to enter only 40 characters

Structural testing involves a detailed technical knowledge of the system.For software, testers create structural tests by looking at the code and the datastructures themselves For hardware, testers create structural tests to comparechip specifications to readings on oscilloscopes or voltage meters Structuraltests thus fit well in the development area Testers in an independent test

team — who often have little exposure to low-level details and might lackprogramming or engineering skills — find it difficult to perform structuraltesting.

Structural tests also involve knowledge of structural testing techniques Notall programmers learn these techniques as part of their initial education andongoing skills growth In such cases, having a member of the test team workwith the programmers as a subject-matter expert can promote good structuraltesting This person can help train the programmers in the techniques needed

to find bugs at a structural level

Behavioral (Black-Box) Tests

Testers use behavioral tests (also known as black-box tests) to find bugs

in high-level operations, such as major features, operational profiles, and

customer scenarios Testers can create black-box functional tests based on what

a system should do For example, if SpeedyWriter should include a featurethat saves files in XML format, then you should test whether it does so Testers

can also create black-box non-functional tests based on how a system should do

what it does For example, if DataRocket can achieve an effective throughput

of only 10 Mbps across two 1-gigabit Ethernet connections acting as a bridge,

a black-box network-performance test can find this bug

Behavioral testing involves a detailed understanding of the applicationdomain, the business problem that the system solves, and the mission the sys-tem serves When testers understand the design of the system, at least at a highlevel, they can augment their behavioral tests to effectively find bugs common

to that type of design For example, programs implemented in languages like

C and C++ can — depending on the programmers’ diligence — suffer fromserious security bugs related to buffer overflows

In addition to the application domain and some of the technological issuessurrounding the system under test, behavioral testers must understand thespecial behavioral test techniques that are most effective at finding suchbugs While some behavioral tests look at typical user scenarios, many testsexercise extremes, interfaces, boundaries, and error conditions Bugs thrive insuch boundaries, and behavioral testing involves searching for defects, just

as structural testing does Good behavioral testers use scripts, requirements,documentation, and testing skills to guide them to these bugs Simply playingaround with the system or demonstrating that the system works under averageconditions are not effective techniques for behavioral testing, although manytest teams make the mistake of adopting these as the sole test techniques Goodbehavioral tests, like good structural tests, are structured, methodical, and oftenrepeatable sequences of tester-created conditions that probe suspected systemweaknesses and strive to find bugs, but through the external interfaces of thesystem under test Most independent test organizations perform primarilybehavioral testing

Live Tests

Live tests involve putting customers, content experts, early adopters, andother end users in front of the system In some cases, we encourage thetesters to try to break the system Beta testing is a well-known form ofbug-driven live testing For example, if the SpeedyWriter product has certainconfiguration-specific bugs, live testing might be the best way to catch thosebugs specific to unusual or obscure configurations In other cases, the testerstry to demonstrate conformance to requirements, as in acceptance testing,another common form of live testing

Live tests can follow general scripts or checklists, but live tests are often

ad hoc (worst case) or exploratory (best case) They don’t focus on systemweaknesses except for the ‘‘error guessing’’ that comes from experience Livetesting is a perfect fit for technical support, marketing, and sales organizationswhose members don’t know formal test techniques but do know the appli-cation domain and the product intimately This understanding, along withrecollections of the nasty bugs that have bitten them before, allows them tofind bugs that developers and testers miss

The Complementary and Continuous Nature of Test Granularity

The crew of a fishing boat uses a tight-mesh net to catch 18-inch salmon and

a loose-mesh net to catch six-foot tuna They might be able to catch a tuna in

a salmon net or vice versa, but it would probably make them less efficient.Likewise, structural, behavioral, and live tests each are most effective at findingcertain types of bugs Many great test efforts include a mix of all three types.While my test teams focus on behavioral testing typically, I don’t feel bound

to declare my test group ‘‘the black-box bunch.’’ I’ve frequently used structuraltest tools and cases effectively as part of my system test efforts I’ve also usedlive production data in system testing Both required advanced planning,but paid off handsomely in terms of efficiency (saved time and effort) andeffectiveness (bugs found that we might have missed) Test granularity is aspectrum, not an either/or categorization Mixing these elements can be useful

in creating test conditions or assessing results I also mix planned test scenarioswith exploratory live testing I use whatever works

A Stampede or a March? Test Phases

The period of test execution activity during development or maintenance issometimes an undifferentiated blob Testing begins, testers run some (vaguelydefined) tests and identify some bugs, and then, at some point, project manage-ment declares testing complete As development and maintenance processesmature, however, companies tend to adopt an approach of partitioning testing