testing applications on the web

Preface xxiIntroduction 4The Evolution of Software Testing 4The Gray-Box Testing Approach 7Real-World Software Testing 9 What’s New in the Second Edition 12 Why Read This Chapter?. 150Va

Hung Q Nguyen

Bob Johnson Michael Hackett

Testing Applications

on the Web: Test Planning for Mobile and

Internet-Based Systems

Second Edition

Hung Q Nguyen

Bob Johnson Michael Hackett

Testing Applications

on the Web: Test Planning for Mobile and

Internet-Based Systems

Second Edition

Executive Publisher: Robert Ipsen

Executive Editor:Carol Long

Development Editor: Scott Amerman

Editorial Manager: Kathryn A Malm

Production Editor: Felicia Robinson

Text Design & Composition:Wiley Composition Services Copyright © 2003 by Hung Q Nguyen, Bob Johnson, and Michael Hackett All rights reserved.

Published by Wiley Publishing, Inc., Indianapolis, Indiana Published simultaneously in Canada

No part of this publication may be reproduced, stored in a retrieval system, or transmitted

in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rose- wood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 646-8700 Requests to the Pub- lisher for permission should be addressed to the Legal Department, Wiley Publishing, Inc.,

10475 Crosspoint Blvd., Indianapolis, IN 46256, (317) 572-3447, fax (317) 572-4447, E-mail: permcoordinator@wiley.com.

Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect

to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose No warranty may

be created or extended by sales representatives or written sales materials The advice and strategies contained herein may not be suitable for your situation You should consult with

a professional where appropriate Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, inci- dental, consequential, or other damages.

For general information on our other products and services please contact our Customer Care Department within the United States at (800) 762-2974, outside the United States at (317) 572-3993 or fax (317) 572-4002.

Trademarks:Wiley, the Wiley Publishing logo and related trade dress are trademarks or registered trademarks of Wiley Publishing, Inc., in the United States and other countries, and may not be used without written permission All other trademarks are the property of their respective owners Wiley Publishing, Inc., is not associated with any product or ven- dor mentioned in this book.

Wiley also publishes its books in a variety of electronic formats Some content that appears

in print may not be available in electronic books.

Library of Congress Cataloging-in-Publication Data:

ISBN: 0-471-20100-6 Printed in the United States of America

10 9 8 7 6 5 4 3 2 1

To Heather, Wendy, Denny, Leilani, Jesse and Anne, whose love and friendship give me the endless source of energy and happiness.

Hung Q Nguyen

To Victoria, for all the advice, help, support, and love she has given me.

Bob Johnson

To Ron, from whom I have stolen much time to make this book happen.

Thank you for your love and support

Michael Hackett

Preface xxi

Introduction 4The Evolution of Software Testing 4The Gray-Box Testing Approach 7Real-World Software Testing 9

What’s New in the Second Edition 12

Why Read This Chapter? 15Introduction 16

Hardware and Software Differences 20The Differences between Web and Traditional

Why Read This Chapter? 41Introduction 42Basic Planning and Documentation 42Common Terminology and Concepts 43

Equivalence Class Partitioning and

Bibliography 80

Why Read This Chapter? 81Introduction 82

Other Useful Information 99

Why Read This Chapter? 111Introduction 112Overview 112

Scripts 123

Web Application Component Architecture 123

Testing Considerations 141

Bibliography 147

Why Read This Chapter? 149Introduction 150What Is a Mobile Web Application? 150Various Types of Mobile Web Client 151

Mobile Web Application Platform

Microbrowsers 159

Other Software Development Platforms

The Device Technology Converging Game:

Bibliography and Additional Resources 172

Bibliography 172

Why Read This Chapter? 177Introduction 178

LogiGear One-Page Test Plan 184

Why Read This Chapter? 193Introduction 194Application Description 194

Functionality of the Sample Application 196

Why Read This Chapter? 203Introduction 204Gathering Information 204

Sample One-Page Test Plan 210Bibliography 212

Part Three Testing Practice 213

Why Read This Chapter? 215Introduction 216User Interface Design Testing 216

User Interface Implementation Testing 243

Usability and Accessibility Testing 247

Testing Considerations 249Bibliography and Additional Resources 251

Bibliography 251

Why Read This Chapter? 253Introduction 254

An Example of Cataloging Features

in Preparation for Functional Tests 254

Why Read This Chapter? 269Introduction 270Common Server-Side Testing Issues 271

Resource Issues 274

Server Side Testing Tips 281

Bibliography 294

Why Read This Chapter? 297Introduction 298Batch or Shell Commands 298

Application of Scripting to Testing Tasks 303

Scripting Project Good Practice 311Scripting Good Practice 312

Perl 314Tcl 315AWK 315

Bibliography and Useful Reading 316

Chapter 14 Database Tests 317

Why Read This Chapter? 317Introduction 318Relational Database Servers 320

Client/SQL Interfacing 325

Database Testing Considerations 349Bibliography and Additional Resources 350

Bibliography 350

Why Read This Chapter? 353Introduction 354

Testing Considerations 365Bibliography 366

Why Read This Chapter? 367Introduction 368The Roles of Installation/Uninstallation Programs 369

Installer 369Uninstaller 371

Common Features and Options 372

Common Server-Side-Specific Installation Issues 384Installer/Uninstaller Testing Utilities 387

Testing Considerations 388Bibliography and Additional Resources 394

Bibliography 394

Why Read This Chapter? 395Introduction 396

Approaching Configuration and Compatibility Testing 398

Comparing Configuration Testing with Compatibility Testing 401Configuration/Compatibility Testing Issues 403

Client-Side Issues 405

Testing Considerations 411Bibliography 414

Why Read This Chapter? 415Introduction 416

Security Solution Basics 424

Cryptography 428

Perimeter-Based Security: Firewalls, DMZs,

Firewalls 432

Common Vulnerabilities and Attacks 435

Backdoors 440JavaScript 440

Java 440ActiveX 441Cookies 441Spoofing 442

Testing Goals and Responsibilities 446

Client Privacy Issues: What Information Needs to Be Private? 451

Backdoors 452

Penetration Testing 463

Other Testing Considerations 473Bibliography and Additional Resources 476

Bibliography 476

Tools 478

Why Read This Chapter? 479Introduction 480Performance Testing Concepts 481

Determining Acceptable Response Time

Three Phases of Performance Testing 493Setting Goals and Expectations

and Defining Deliverables 494Gathering Requirements 496

Defining the Workload 497

Problems Concerning Workloads 504Selecting Performance Metrics 505

Which Tests to Run and When to Start 508Tool Options and Generating Loads 512

Writing the Test Plan 515

Identifying Baseline Configuration

Determine Whether the Testing Process Will Be

Other Testing Considerations 523Bibliography 525

Why Read This Chapter? 527Introduction 528Testing Mobile versus Desktop Web Applications 528Various Types of Tests 536

Latency 541

Testing Web Applications Using

Testing Web Applications Using

Survey of Mobile Testing Support Tools 546

OpenWave 547Nokia 548YoSpace 548Microsoft 548Web-Based Mobile Phone Emulators

Other Testing Considerations 549Bibliography and Additional Resources 550

Bibliography 550

Why Read This Chapter? 553Introduction 554

Sample List of Rule-Based Analyzers for C/C++, Java, Visual Basic, and Other

Sample List of Automated GUI Functional

QACity.Com Comprehensive List of DEFECT TRACKING

Why Read This Chapter? 567Introduction 568Textbooks 568

Miscellaneous Papers on the Web from Carnegie Mellon

Professional Societies 576

Appendix A LogiGear Test Plan Template 579

Appendix D UI Test-Case Design Guideline: Common Keyboard

Appendix F Web Test-Case Design Guideline: Input Boundary

Testing Applications on the Web introduces the essential technologies, testing

concepts, and techniques that are associated with browser-based applications

It offers advice pertaining to the testing of business-to-business applications,business-to-end-user applications, Web portals, and other Internet-based appli-cations The primary audience is software testers, software quality engineers,quality assurance staff, test managers, project managers, IT managers, busi-ness and system analysts, and anyone who has the responsibility of planningand managing Web-application test projects

Testing Applications on the Web begins with an introduction to the

client-server and Web system architectures It offers an in-depth exploration of Webapplication technologies such as network protocols, component-based archi-tectures, and multiple server types from the testing perspective It then coverstesting practices in the context of various test types from user interface tests toperformance, load, and stress tests, and security tests Chapters 1 and 2 present

an overview of Web testing Chapters 3 through 6 cover methodology andtechnology basics, including a review of software testing basics, a discussion

on networking, an introduction to component-based testing, and an overview

of the mobile device platform Chapters 7 through 9 discuss testing planningfundamentals, a sample application to be used as an application under test(AUT) throughout the book, and a sample test plan Chapters 10 through 20discuss test types that can be applied to Web testing Finally, Chapters 21 and

22 offer a survey of Web testing tools and suggest where to go for additionalinformation

Testing Applications on the Web answers testing questions such as, “How do

networking hardware and software affect applications under test?” “What areWeb application components, and how do they affect my testing strategies?”

Preface

xxi

“What is the role of a back-end database, and how do I test for related errors?” “How do I test server-side software?” “What are performance,stress, and load tests, and how do I plan for and execute them?” “What do Ineed to know about security testing, and what are my testing responsibili-ties?” “What do I need to consider in testing mobile Web applications?”With a combination of general testing methodologies and the informationcontained in this book, you will have the foundation required to achieve thesetesting goals—maximizing productivity and minimizing quality risks in aWeb application environment.

database-Testing Applications on the Web assumes that you already have a basic

under-standing of software testing methodologies, including test planning, test-casedesign, and bug report writing Web applications are complex systems thatinvolve numerous components: servers, browsers, third-party software andhardware, protocols, connectivity, and much more This book enables you toapply your existing testing skills to the testing of Web applications

N OT E This book is not an introduction to software testing If you are looking for fundamental software testing practices, you will be better served by reading

Testing Computer Software,Second Edition, by Kaner, Cem, Jack Falk, and Hung Q Nguyen (Wiley, 1999) For additional information on Web testing and other testing techniques and resources, visit www.QAcity.com.

We have enjoyed writing this book and teaching the Web application testingtechniques that we use every day to test Web-based systems We hope that youwill find here the information you need to plan for and execute a successfultesting strategy that enables you to deliver high-quality applications in anincreasingly distributed-computing, market-driven, and time-constrainedenvironment in this era of new technology

Writing about Web testing is challenging because the field involves the dependence of so many different technologies and systems It’s not enough

inter-to write about the client Certainly, the client software is the part of the cation that is the most visible to the customer, and it’s the easiest to write about(authors can just repackage the same old stuff published about applications

appli-in general Hung, Michael, and Bob do provide client-side guidance, but theirgoal is to provide information that is specific to Web applications (For more

generic material, you can read Testing Computer Software, Second Edition,

Wiley, 1999.)But client-side software is just the tip of the iceberg The application dis-plays itself to the end user as the client, but it does most of its work in con-junction with other software on the server-side, much of it written andmaintained by third parties For example, the application probably stores andretrieves data via third-party databases If it sells products or services, it prob-ably clears customer orders with the customer’s credit card company It mightalso check its distributor for available inventory and its shippers for the cost ofshipping the software to the customer The Web application communicateswith these third parties through network connections written by third parties.Even the user interface is only partially under the application developer’s control—the customer supplies the presentation layer: the browser, the musicand video player, and perhaps various other multimedia plug-ins

The Web application runs on a broader collection of hardware and softwareplatforms than any other type of application in history Attributes of these plat-forms can change at any time, entirely outside of the knowledge or control ofthe Web application developer

Foreword

xxiii

In Testing Applications on the Web, Nguyen, Hackett, and Johnson take this

complexity seriously In their view, a competent Web application tester mustlearn the technical details of the systems with which the application under testinteracts To facilitate this, they survey many of those systems, explaining howapplications interact with them and providing testing tips

As a by-product of helping testers appreciate the complexity of the Web

test-ing problem, the first edition of Testtest-ing Applications on the Web became the first

book on gray-box testing In so-called black-box testing, we treat the softwareunder test as a black box We specify the inputs, we look at the outputs, but wecan’t see inside the box to see how it works The black-box tester operates at

the customer’s level, basing tests on knowledge of how the system should

work In contrast, the white-box tester knows the internals of the software, anddesigns tests with direct reference to the program’s source code The gray-boxtester doesn’t have access to the source code, but he or she knows much moreabout the underlying architecture and the nature of the interfaces between theapplication under test and the other software and the operating systems.The second edition continues the gray-box analysis by deepening the dis-cussions in the first edition It also adds several new chapters to address business-critical testing issues from server-side, performance- and application-level security testing to the latest mobile Web application testing A finalstrength of the book is the power of the real-world example Hung QuocNguyen is the president of the company that published TRACKGEAR, a Web-based bug tracking system, enabling the authors can give us the inside story ofits development and testing

This combination of a thorough and original presentation of a style of sis, mixed with detailed insider knowledge is a real treat to read It teaches usabout thinking through the issues involved when the software under testinteracts in complex ways with many other programs, and it gives the book avalue that will last well beyond the specifics of the technologies describedtherein

analy-Cem Kaner, J.D., Ph D

Professor of Computer SciencesFlorida Institute of Technology

While it is our names that appear on the cover, over the years, many peoplehave helped with the development of this book We want to particularly thankBrian Lawrence, for his dedication in providing thorough reviews and criticalfeedback We all thank Cem Kaner for his guidance, friendship, and generosity,and for being there when we needed him We thank, too, Jesse Watkins-Gibbsfor his work on examples and sample code, as well as for his technical exper-tise and his commitment to getting our book done

We would also like to thank our professional friends who took time outfrom their demanding jobs and lives to review and add comment on the book:Yannick Bertolus, George Hamblin, Elisabeth Hendrickson, NematolahKashanian, Pat McGee, Alberto Savoia, and Garrin Wong We would like tothank our copyeditor Janice Borzendowski We also want to thank the follow-ing people for their contributions (listed in alphabetical order): James L Carr,William Coleman, Norm Hardy, Pam Hardy, Thomas Heinz, Chris Hibbert,Heather Ho, Brian Jones, Denny Nguyen, Kevin Nguyen, Wendy Nguyen,Steve Schuster, Kurt Thams, Anne Tran, Dean Tribble, and Joe Vallejo Finally,

we would like to thank our colleagues, students, and staff at LogiGear ration, for their discussions and evaluations of the Web testing training mate-rial, which made its way into this book And thanks to our agent ClaudetteMoore of Moore Literacy Agency

Corpo-Certainly, any remaining errors in the book are ours

Acknowledgments

xxv

Hung Q Nguyenis Founder, President, and CEO of LogiGear Corporation.Nguyen has held leadership roles in business management, product develop-ment, business development, engineering, quality assurance, software testing,and information technology Hung is an international speaker and a regularcontributor to industry publications He is the original architect of TRACK-GEAR, a Web-based defect management system Hung also teaches softwaretesting for the University of California at Berkeley and Santa Cruz Extension,and LogiGear University.

Hung is the author of Testing Applications on the Web, First Edition (Wiley 2000); and with Cem Kaner and Jack Falk, he wrote the best-selling book Test-

ing Computer Software (ITP/Wiley 1993/1999) He holds a Bachelor of Science

in Quality Assurance from Cogswell Polytechnical College, and is an Certified Quality Engineer and a member of the Advisory Council for theDepartment of Applied Computing and Information Systems at UC BerkeleyExtension

ASQ-You can reach Hung at hungn@logigear.com; or, to obtain more informationabout LogiGear Corporation and Hung’s work, visit www.logigear.com

Bob Johnson has been a software developer, tester, and manager of bothdevelopment and testing organizations With over 20 years of experience insoftware engineering, Bob has acquired key strengths in building applications

on a variety of platforms Bob’s career in software development ranges fromWeb programming to consulting on legal aspects of e-commerce to the require-ment and review process Whether working in test automation, Web security, orback-end server testing, Bob is at the forefront of emerging technologies

About the Authors

xxvii

In addition to participating in the Los Altos Workshops on Software Testing

(LAWST), Bob has written articles for IEEE Software, Journal of Electronic

Commerce, and Software Testing and Quality Engineering He can be reached at

rjohnson@testingbook.com

Michael Hackett isVice President and a founding partner of LogiGear ration He has over a decade of experience in software engineering and thetesting of shrink-wrap and Internet-based applications Michael has helpedwell-known companies release applications ranging from business productiv-ity to educational multimedia titles, in English as well as a multitude of otherlanguages Michael has taught software testing for the University of California

Corpo-at Berkeley Extension, the Software Productivity Center in Vancouver, theHong Kong Productivity Centre, and LogiGear University Michael holds aBachelor of Science in Engineering from Carnegie-Mellon University He can

be reached at michaelh@logigear.com

PA R T

One Introduction

C H A P T E R

1

Why Read This Chapter?

The goal of this book is to help you effectively plan for and conduct the testing

of Web-based applications developed for fixed clients, systems in fixed tions such as desktop computers, as well as for mobile clients such as mobilephones, PDAs (Personal Digital Assistants), and portable computers Thisbook will be more helpful to you if you understand the philosophy behind itsdesign

loca-Software testing practices have been improving steadily over the past fewdecades Yet, as testers, we still face many of the same challenges that we havefaced for years We are challenged by rapidly evolving technologies and theneed to improve testing techniques We are also challenged by the lack ofresearch on how to test for and analyze software errors from their behavior, asopposed to at the source code level We are challenged by the lack of technicalinformation and training programs geared toward serving the growing popu-lation of the not-yet-well-defined software testing profession Finally, we arechallenged by limited executive management support as a result of manage-ment’s underestimation and lack of attention to the cost of quality Yet, in

Welcome to Web

Testing

C H A P T E R

This chapter offers a historical perspective on the changing objectives of ware testing It touches on the gray-box testing approach and suggests theimportance of having a balance of product design, both from the designer’sand the user’s perspective, and system-specific technical knowledge It alsoexplores the value of problem analysis to determine what to test, when to test,and where to test Finally, this chapter will discuss what assumptions this bookhas about the reader

soft-The Evolution of Software Testing

As the complexities of software development have evolved over the years, thedemands placed on software engineering, information technology (IT), andsoftware quality professionals have grown and taken on greater relevance Weare expected to check whether the software performs in accordance with itsintended design and to uncover potential problems that might not have beenanticipated in the design We are expected to develop and execute more tests,faster, and more often Test groups are expected to offer continuous assess-ment on the current state of the projects under development At any givenmoment, we must be prepared to report explicit details of testing coverage andstatus, the health or stability of the current release, and all unresolved errors.Beyond that, testers are expected to act as user advocates This often involves

TOPICS COVERED IN THIS CHAPTER

◆ Introduction

◆ The Evolution of Software Testing

◆ The Gray-Box Testing Approach

◆ Real-World Software Testing

◆ Themes of This Book

◆ What’s New in the Second Edition

today’s world of Internet time, the systems under test are getting more complex

by the day, and resources and testing time are in short supply The quicker wecan get the information that we need, the more productive and more success-ful we will be at doing our job The goal of this book is to help you do your jobeffectively

anticipating usability problems early in the development process so thoseproblems can be addressed in a timely manner.

In the early years, on mainframe systems, many users were connected to acentral system Bug fixing involved patching or updating the centrally storedprogram This single fix would serve the needs of hundreds or thousands ofindividuals who used the system

As computing became more decentralized, minicomputers and puters were run as stand-alone systems or on smaller networks There weremany independent computers or local area networks, and a patch to the code

microcom-on microcom-one of these computers updated relatively fewer people Mass-market ware companies sometimes spent over a million dollars sending disks to reg-istered customers just to fix a serious defect Additionally, technical supportcosts skyrocketed

soft-As the market has broadened, more people use computers for more thingsand rely more heavily on computers, hence the consequences of softwaredefects has risen every year It is impossible to find all possible problems bytesting, but as the cost of failure has gone up, it has become essential to do risk-based testing In a risk-based approach, you ask questions like these:

■■ Which areas of the product are so significant to the customer or soprone to serious failure that they must be tested with extreme care?

■■ For the average area, and for the program as a whole, how much testing

is enough?

■■ What are the risks involved in leaving a certain bug unresolved?

■■ Are certain components so unimportant as to not merit testing?

■■ At what point can a product be considered adequately tested and readyfor market?

■■ How much longer can the product be delayed for testing and fixingbugs before the market viability diminishes the return on investment?

Tracking bugs, analyzing and assessing their significance are priorities.Management teams expect development and IT teams, testing and qualityassurance staff, to provide quantitative data regarding test coverage, the status

of unresolved defects, and the potential impact of deferring certain defects Tomeet these needs, testers must understand the products and technologies theytest They need models to communicate assessments of how much testing hasbeen done in a given product, how deep testing will go, and at what point theproduct will be considered adequately tested Given better understanding oftesting information, we make better predictions about quality risks

In the era of the Internet, the connectivity that was lost when computingmoved from the mainframe model to the personal computer (PC) model,

in effect, has been reestablished Personal computers are effectively networkedover the Internet Bug fixes and updated builds are made available—sometimes on a daily basis—for immediate download over the Internet.Product features that are not ready by ship date are made available later in

service packs The ability to distribute software over the Internet has brought

down much of the cost that is associated with distributing some applicationsand their subsequent bug fixes

Although the Internet offers connectivity for PCs, it does not offer the trol over the client environment that was available in the mainframe model.The development and testing challenges with the Graphical User Interface(GUI) and event-based processing of the PC are enormous because the clientsattempt remarkably complex tasks on operating systems (OSs) as different asUNIX, Macintosh OS, Linux, and the Microsoft OSs They run countless com-binations of processors, peripherals, and application software Additionally,the testing of an enterprise client-server system may require the consideration

con-of thousands con-of combinations con-of OSs, modems, routers, and client-serversoftware components and packages Web applications increase this complex-ity further by introducing browsers and Web servers into the mix Further-more, wireless networks are becoming more pervasive, and their bandwidthscontinue to improve On the client-side, computer engineers continue toadvance in building smaller, yet more powerful portable or mobile devices.Communication and wearable devices and Internet appliances are expandingthe possible combinations of Web client environments beyond the desktopenvironments On the server-side, software components that were normallylocated in a company’s enterprise server are making their move toward theapplication services or Web services model In this model, the components will

be hosted outside of the corporate enterprise server, usually at the third-partyASPs (application service providers), adding more challenges to the testing ofInternet-based systems

Software testing plays a more prominent role in the software developmentprocess than ever before Developers are paying more attention to buildingtestability into their code, as well as coming up with more ways to improve theunit-test framework around their production code Companies are allocatingmore money and resources for testing because they understand that their rep-utations and success rest on the quality of their products, or that their failure ismore probable due to poor product and service quality The competitiveness ofthe computing industry (not to mention the savvy of most computer users) haseliminated most tolerance for buggy software Nevertheless, many companies

believe that the only way to compete in Internet time is to develop software as

rapidly as possible Short-term competitive issues often outweigh qualityissues One consequence of today’s accelerated development schedules is the

industry’s tendency to push software out into the marketplace as early as sible Development teams get less and less time to design, code, test, andundertake process improvements Market constraints and short developmentcycles often do not allow time for reflection on past experience and considera-tion of more efficient ways to produce and test software.

pos-The Gray-Box Testing Approach

Black-box testing focuses on software’s external attributes and behavior Suchtesting looks at an application’s expected behavior from the user’s point ofview White-box testing (also known as glass-box testing), on the other end ofthe spectrum, tests software with knowledge of internal data structures, phys-ical logic flow, and architecture at the source code level White-box testinglooks at testing from the developer’s point of view Both black-box and white-box testing are critically important complements of a complete testing effort.Individually, they do not allow for balanced testing Black-box testing can beless effective at uncovering certain error types, such as data-flow errors orboundary condition errors at the source level White-box testing does not read-ily highlight macro-level quality risks in operating environment, compatibil-ity, time-related errors, and usability

Gray-box testing incorporates elements of both black-box and white-boxtesting It considers the outcome on the user end, system-specific technicalknowledge, and operating environment It evaluates application design in the

context of the interoperability of system components The gray-box testing

approach is integral to the effective testing of Web applications because Webapplications comprise numerous components, both software and hardware.These components must be tested in the context of system design to evaluatetheir functionality and compatibility

In our view, gray-box testing consists of methods and tools derived from theknowledge of the application internals and the environment with which itinteracts The knowledge of the designer’s intended logic can be applied intest design and bug analysis to improve the probability of finding and repro-ducing bugs See Chapter 5, “Web Application Components,” the section enti-tled “Testing Discussion,” for an example of gray-box testing methods

Here are several unofficial definitions for gray-box testing from the LosAltos Workshop on Software Testing (LAWST) IX (For more information onLAWST, visit www.kaner.com.)

Gray-box testing—Using inferred or incomplete structural or design mation to expand or focus black-box testing —Dick Bender

infor-Gray-box testing—Tests designed based on the knowledge of algorithms,internal states, architectures, or other high-level descriptions of programbehavior —Doug Hoffman

Gray-box testing—Tests involving inputs and outputs; but test design iseducated by information about the code or the program operation of akind that would normally be out of scope of the view of the tester —Cem Kaner

Gray-box testing is well suited for Web application testing because it factors

in high-level design, environment, and interoperability conditions It willreveal problems that are not as easily considered by a black-box or white-boxanalysis, especially problems of end-to-end information flow and distributedhardware/software system configuration and compatibility Context-specificerrors that are germane to Web systems are commonly uncovered in thisprocess

Another point to consider is that many of the types of errors that we run into

in Web applications might well be discovered by black-box testers, if only wehad a better model of the types of failures for which to look and design tests.Unfortunately, we are still developing a better understanding of the risks thatare associated with the new application and communication architectures

Therefore, the wisdom of traditional books on testing (e.g., Testing Computer

Software, 2nd ed., John Wiley & Sons, Inc (1999), by Kaner, Falk, and Nguyen

and Lessons Learned in Software Testing, John Wiley & Sons, Inc (2002), by

Kaner, Bach, and Pettichord) will not fully prepare the black-box tester tosearch for these types of errors If we are equipped with a better under-standing of the system as a whole, we’ll have an advantage in exploring thesystem for errors and in recognizing new problems or new variations of olderproblems

As testers, we get ideas for test cases from a wide range of knowledge areas.This is partially because testing is much more effective when we know andmodel after the types of bugs we are looking for We develop ideas of whatmight fail, and of how to find and recognize such a failure, from knowledge ofmany types of things [e.g., knowledge of the application and system architec-ture, the requirements and use of this type of application (domain expertise),and software development and integration] As testers of complex systems, weshould strive to attain a broad balance in our knowledge, learning enoughabout many aspects of the software and systems being tested to create a bat-tery of tests that can challenge the software as deeply as it will be challenged

in the rough and tumble of day-to-day use

Finally, we are not suggesting that every tester in a group be a gray-box

tester We have seen a high level of success in several test teams that have a mix

of different types of testers, with different skill sets (e.g., subject matter expert,

database expert, security expert, API testing expert, test automation expert,etc.) The key is, within that mix, for at least some of the testers to understandthe system as a collection of components that can fail in their interaction witheach other, and these individuals must understand how to control and how tosee those interactions in the testing and production environments.

Real-World Software Testing

Web businesses have the potential to be high-profit ventures In the dot-comera, venture capitalists could support a number of losing companies as long asthey had a few winners to make up for their losses A CEO has three to fiveyears to get a start-up ready for IPO (six months to prove that the prototypeworks, one or two years to generate some revenue—hence, justifying the busi-ness model—and the remainder of the time to show that the business can beprofitable someday) It is always a challenge to find enough time and quali-fied personnel to develop and deliver quality products in such a fast-pacedenvironment

Although standard software development methodologies such as ity Maturity Model (CMM) and ISO-9000 have been available, they are not yetwell accepted by aggressive start-up companies These standards and meth-ods are great practices, but the fact remains that many companies will rely onthe efforts of a skilled development and testing staff, rather than a process thatthey fear might slow them down In that situation, no amount of improvedstandards and process efficiencies can make up for the efforts of a skilleddevelopment and testing staff That is, given the time and resource constraints,they still need to figure out how to produce quality software

Capabil-The main challenge that we face in Web application testing is to learn theassociated technologies in order to have a better command over the environ-ment We need to know how Web technologies affect the interoperability ofsoftware components, as well as Web systems as a whole Testers also need toknow how to approach the testing of Web-based applications This requiresbeing familiar with test types, testing issues, common software errors, and thequality-related risks that are specific to Web applications We need to learn,and we need to learn fast Only with a solid understanding of software testingbasics and a thorough knowledge of Web technologies can we competentlytest Web-based systems

The era of high-profit dot-com ventures has ended Many businesses based

on the dot-com model have closed their doors The NASDAQ Stock Exchangewithin three years of March 2000, from its best performance of above 5,000points, has dropped to as low as 1,100 We have learned many great and

painful business lessons from this era, but unfortunately, not many related lessons However, the good news is, in a difficult economic time, themarket demand is low, hence customers are in control They want to pay lessmoney for more, and demand higher-quality products and services Bugs left

quality-in the product could mean anythquality-ing from high support costs to loss of sales

to contract cancellation That is money loss that companies can normallyabsorb during the good economic times, but not in the bad times We haveseen many movements and initiatives in which executive staffs are payingmore attention and asking more interesting quality-related questions They areasking questions like, “How can we produce better software (higher-quality),faster (improve time-to-market) at a lower cost (lower production and qualitycosts)?” While this type of question has been asked before, executive staffsnow have more time to listen and are more willing to support the QA effort Italso means a lot more demand is placed on testing, which requires QA staff to

be more skilled in testing strategy, better educated in software engineering andtechnology, better equipped with testing practices, and savvier in the conceptand application of software testing tools

Themes of This Book

The objective of this book is to introduce testers into the discipline of gray-boxtesting, by offering readers information about the interplay of Web applica-tions, component architectural designs, and their network systems We expectthat this will help testers develop new testing ideas, enabling them to uncoverand troubleshoot new types of errors and conduct more effective root-causeanalyses of software failures discovered during testing or product use Thediscussions in this book focus on determining what to test, where to test, andwhen to test As appropriate, real-world testing experiences and examples oferrors are included

To effectively plan and execute the testing of your Web application, youneed to possess the following qualities: good software testing skill; knowledge

of your application, which you will need to provide; knowledge of Web nologies; understanding of the types of tests and their applicability to Webapplication; knowledge of several types of Web application-specific errors (soyou know what to look for); and knowledge of some of the available tools andtheir applicability, which this book offers you (See Figure 1.1.)

tech-Based on this knowledge and skill set, you can analyze the testing ments to come up with an effective plan for your test execution If this is whatyou are looking for, this book is for you It is assumed that you have a solidgrasp of standard software testing practices and procedures