www.it-ebooks.info “Great book! Whittaker delivers ideas that are innovative, smart, and mem- orable. He really knows how to inspire engineers to think differently about testing.” —Patrick Copeland, Director of Test Engineering, Google “James has perfected a fantastic manual testing methodology. The touring concept not only works, but it works so well that we’ve started sharing the tour concepts in the internal testing courses taught to all of our testers. If you want to bring your manual testing processes into the 21st century then read this book.” —Alan Page, Director, Test Excellence, Microsoft “I began working with James at IBM in 1990. Even then, he was inspiring testers and developers to think outside the box. With this book he’s taken his passion for software quality to a whole new level. Read it and watch yourself become a better tester. James is the real deal and this book should be read by every tester and software developer on the planet who cares about software quality or just wants to have more fun doing what they do.” —Kaushal K. Agrawal, Sr. Director of Engineering, Cisco Systems “James Whitaker is a true visionary in the world of testing. uTest and our global community of QA professionals regularly look to James for inspira- tion, interpretation of trends, and overarching testing wisdom. Now he’s finally written it down for everyone else and our industry will be smarter because of it.” —Doron Reuveni, CEO and Co-Founder, uTest “Only someone like James Whittaker would think of combining the idea of tourism with software testing in such a novel way—and only James could pull it off. The tours approach provides a memorable and extremely effec- tive mental model that combines right degree of structure and organization with plenty of room for exploration and creativity. Bugs beware!” —Alberto Savoia, Google “James is one of the best speakers around on software testing and reading his book is much like hearing him speak. If you want to increase your knowledge of testing and make yourself a better tester, this is the book for you.” —Stewart Noakes, Chairman and Co-Founder, TCL Group Ltd. www.it-ebooks.info “I’ve been doing exploratory testing for some time now and James’ tours have given what I do a name, a focus and more importantly some actual guidance. This book is going to make the job of teaching and performing exploratory testing a whole lot easier.” —Rob Lambert, Senior Test Consultant, iMeta Technologies Ltd “I’m pretty pumped up about this work—it’s sane, it’s new, and I, a normal human, can understand and use it without first studying the combined works of various pompous, dead philosophers. I didn’t have to resort to a dictionary once in the chapters I read. I genuinely feel this work is at the forefront of some long-awaited and sorely-needed evolution for our field.” —Linda Wilkinson, QA Manager, NetJets, Inc. www.it-ebooks.info Exploratory Software Testing www.it-ebooks.info This page intentionally left blank www.it-ebooks.info James A. Whittaker Upper Saddle River, NJ • Boston • Indianapolis • San Francisco New York • Toronto • Montreal • London • Munich • Paris • Madrid Capetown • Sydney • Tokyo • Singapore • Mexico City Exploratory Software Testing www.it-ebooks.info Many of the designations used by manufacturers and sellers to distin- guish their products are claimed as trademarks. Where those designa- tions appear in this book, and the publisher was aware of a trademark claim, the designations have been printed with initial capital letters or in all capitals. The author and publisher have taken care in the preparation of this book, but make no expressed or implied warranty of any kind and assume no responsibility for errors or omissions. No liability is assumed for incidental or consequential damages in connection with or arising out of the use of the information or programs contained herein. The publisher offers excellent discounts on this book when ordered in quantity for bulk purchases or special sales, which may include elec- tronic versions and/or custom covers and content particular to your business, training goals, marketing focus, and branding interests. For more information, please contact: U.S. Corporate and Government Sales (800) 382-3419 corpsales@pearsontechgroup.com For sales outside the United States please contact: International Sales international@pearson.com Visit us on the Web: informit.com/aw Library of Congress Cataloging-in-Publication Data: Whittaker, James A., 1965- Exploratory software testing : tips, tricks, tours, and techniques to guide manual testers / James A. Whittaker. — 1st ed. p. cm. ISBN 978-0-321-63641-6 (pbk. : alk. paper) 1. Computer software— Testing. I. Title. QA76.76.T48W465 2009 005.1’4—dc22 2009023290 Copyright © 2010 Pearson Education, Inc. All rights reserved. Printed in the United States of America. This pub- lication is protected by copyright, and permission must be obtained from the publisher prior to any prohibited reproduction, storage in a retrieval system, or transmission in any form or by any means, elec- tronic, mechanical, photocopying, recording, or likewise. For informa- tion regarding permissions, write to: Pearson Education, Inc Rights and Contracts Department 501 Boylston Street, Suite 900 Boston, MA 02116 Fax (617) 671 3447 ISBN-13: 978-0-321-63641-6 ISBN-10: 0-321-63641-4 Text printed in the United States on recycled paper at Courier in Stoughton, Massachusetts. First printing August 2009 Editor-in-Chief Karen Gettman Acquisitions Editor Chris Guzikowski Development Editor Mark Renfrow Managing Editor Kristy Hart Senior Project Editor Lori Lyons Copy Editor Keith Cline Indexer Tim Wright Proofreader Apostrophe Editing Services Publishing Coordinator Raina Chrobak Cover Designer Alan Clements Senior Compositor Gloria Schurick www.it-ebooks.info This book was written in large part while I was an architect at Microsoft. It is dedicated to all the talented testers who crossed my path while I was there. Thanks, you changed the way I thought, worked, and envisioned the discipline of software testing. Keep up the good work! www.it-ebooks.info This page intentionally left blank www.it-ebooks.info A Fault to Guide Software Testing 010010101011011000100100100101010110110001001001001010101 Table of Contents Foreword by Alan Page xv Preface xvii Chapter 1 The Case for Software Quality 1 The Magic of Software 1 The Failure of Software 4 Conclusion 9 Exercises 9 Chapter 2 The Case for Manual Testing 11 The Origin of Software Bugs 11 Preventing and Detecting Bugs 12 Preventing Bugs 12 Detecting Bugs 13 Manual Testing 14 Scripted Manual Testing 15 Exploratory Testing 16 Conclusion 19 Exercises 20 Chapter 3 Exploratory Testing in the Small 21 So You Want to Test Software? 21 Testing Is About Varying Things 23 User Input 23 What You Need to Know About User Input 24 How to Test User Input 25 State 32 What You Need to Know About Software State 32 How to Test Software State 33 Code Paths 35 User Data 36 Environment 36 Conclusion 37 Exercises 38 www.it-ebooks.info [...]... Scenario-Based Exploratory Testing Strategy-Based Exploratory Testing Feedback-Based Exploratory Testing www.it-ebooks.info xiii 148 148 149 151 152 153 154 154 155 156 156 157 160 160 161 161 165 165 166 166 167 168 169 170 171 173 174 175 176 177 178 179 180 183 183 183 184 184 xiv Contents The Future of Testing (Part 1) The Future of Testing (Part 2) September 2008 On Certification The Future of Testing. .. Future of Testing (Part 4) The Future of Testing (Part 5) October 2008 The Future of Testing (Part 6) The Future of Testing (Part 7) The Future of Testing (Part 8) Speaking of Google Manual Versus Automated Testing Again November 2008 Software Tester Wanted Keeping Testers in Test December 2008 Google Versus Microsoft and the Dev:Test Ratio Debate January 2009 The Zune Issue Exploratory Testing Explained... and Software Testing) Measuring Testers Prevention Versus Cure (Part 1) Users and Johns Ode to the Manual Tester Prevention Versus Cure (Part 2) Hail Europe! The Poetry of Testing Prevention Versus Cure (Part 3) Back to Testing August 2008 Prevention Versus Cure (Part 4) If Microsoft Is So Good at Testing, Why Does Your Software Still Suck? Prevention Versus Cure (Part 5) Freestyle Exploratory Testing. .. important, mission-critical software Finally, I end the book with two chapters aimed at wrapping up the information from earlier chapters In Chapter 7, “Touring and Testing s Primary Pain Points,” I describe what I see as the hardest problems in testing and how purposeful exploratory testing fits into the broader solutions www.it-ebooks.info Preface In Chapter 8, “The Future of Software Testing, ” I look further... but also for their future contributions to exploratory testing at Google www.it-ebooks.info A Fault to Guide Software Testing 010010101011011000100100100101010110110001001001001010101 About the Author James Whittaker has spent his career in software testing and has left his mark on many aspects of the discipline He was a pioneer in the field of model-based testing, where his Ph.D dissertation from the... test cases that makes the software work as hard as possible and thus find as many important bugs as possible? These are overarching issues of test case design and test suite quality that have to be addressed In Chapter 5, “Hybrid Exploratory Testing Techniques,” the concept of tours is taken a step further by combining exploratory testing with traditional script or scenario-based testing We discuss ways... Chapter 8 The Future of Software Testing Welcome to the Future The Heads-Up Display for Testers “Testipedia” Test Case Reuse Test Atoms and Test Molecules Virtualization of Test Assets Visualization Testing in the Future Post-Release Testing Conclusion Exercises 123 123 124 126 127 128 129 129 132 134 134 135 Appendix A Building a Successful Career in Testing How Did You Get into Testing? Back to the... 2, “The Case for Manual Testing, ” I make the point that because users find these bugs while using the software, testing must also use the software to find them For automation, unit testing, and so forth, these bugs are simply inaccessible Automate all you want, these bugs will defy you and resurface to plague your users The problem is that much of the modern practice of manual testing is aimless, ad... www.it-ebooks.info 7 8 Exploratory Software Testing And then there are the small inconveniences, the things that make you go hmm, as shown in Figure 1.5 This is a montage of various error messages often displayed to show just how confused the software is or as the last gasp of the software before it dies completely I am not the only tester I know who collects these as reminders of why testing is so important... and experiment with search terms similar to software failure.” See whether you can find examples of the following: a The software of a famous vendor failed catastrophically b A software failure caused loss of human life c It was argued that software threatened the American democracy d The financial costs attributed to software failure exceed $1 billion e A software failure affected more than 10,000 . 183 Freestyle Exploratory Testing 183 Scenario-Based Exploratory Testing 183 Strategy-Based Exploratory Testing 184 Feedback-Based Exploratory Testing 184 Contents xiii www.it-ebooks.info The Future of Testing. Bugs 13 Manual Testing 14 Scripted Manual Testing 15 Exploratory Testing 16 Conclusion 19 Exercises 20 Chapter 3 Exploratory Testing in the Small 21 So You Want to Test Software? 21 Testing Is About. xvii Chapter 1 The Case for Software Quality 1 The Magic of Software 1 The Failure of Software 4 Conclusion 9 Exercises 9 Chapter 2 The Case for Manual Testing 11 The Origin of Software Bugs 11 Preventing