Testing Applications on the Web: Test Planning for Mobile and Internet-Based Systems pdf

Testing Considerations 141Script Testing Issues 143Characteristics of a Script 143Use of Scripts in Web Applications 144Testing Scripts in Web Applications 145Coding-Related Problems 145

Hung Q Nguyen

Bob Johnson Michael Hackett

Testing Applications

on the Web: Test Planning for Mobile and

Internet-Based Systems

Second Edition

Hung Q Nguyen

Bob Johnson Michael Hackett

Testing Applications

on the Web: Test Planning for Mobile and

Internet-Based Systems

Second Edition

Executive Publisher: Robert Ipsen

Executive Editor:Carol Long

Development Editor: Scott Amerman

Editorial Manager: Kathryn A Malm

Production Editor: Felicia Robinson

Text Design & Composition:Wiley Composition Services Copyright © 2003 by Hung Q Nguyen, Bob Johnson, and Michael Hackett All rights reserved.

Published by Wiley Publishing, Inc., Indianapolis, Indiana Published simultaneously in Canada

No part of this publication may be reproduced, stored in a retrieval system, or transmitted

in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rose- wood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 646-8700 Requests to the Pub- lisher for permission should be addressed to the Legal Department, Wiley Publishing, Inc.,

10475 Crosspoint Blvd., Indianapolis, IN 46256, (317) 572-3447, fax (317) 572-4447, E-mail: permcoordinator@wiley.com.

Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect

to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose No warranty may

be created or extended by sales representatives or written sales materials The advice and strategies contained herein may not be suitable for your situation You should consult with

a professional where appropriate Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, inci- dental, consequential, or other damages.

For general information on our other products and services please contact our Customer Care Department within the United States at (800) 762-2974, outside the United States at (317) 572-3993 or fax (317) 572-4002.

Trademarks:Wiley, the Wiley Publishing logo and related trade dress are trademarks or registered trademarks of Wiley Publishing, Inc., in the United States and other countries, and may not be used without written permission All other trademarks are the property of their respective owners Wiley Publishing, Inc., is not associated with any product or ven- dor mentioned in this book.

Wiley also publishes its books in a variety of electronic formats Some content that appears

in print may not be available in electronic books.

Library of Congress Cataloging-in-Publication Data:

ISBN: 0-471-20100-6 Printed in the United States of America

10 9 8 7 6 5 4 3 2 1

To Heather, Wendy, Denny, Leilani, Jesse and Anne, whose love and friendship give me the endless source of energy and happiness.

Hung Q Nguyen

To Victoria, for all the advice, help, support, and love she has given me.

Bob Johnson

To Ron, from whom I have stolen much time to make this book happen.

Thank you for your love and support

Michael Hackett

Preface xxi

Introduction 4The Evolution of Software Testing 4

What’s New in the Second Edition 12

New Contents and Significant Updates 12What Remains from the First Edition 13

Introduction 16

Hardware and Software Differences 20The Differences between Web and Traditional

Web Systems 28

Server-Based Applications 31Distributed Server Configurations 32

Thin-Client versus Thick-Client Processing 35

Bibliography 38

Introduction 42Basic Planning and Documentation 42Common Terminology and Concepts 43

The Combinatorial Method 78

Bibliography 80

Direct Connection 86Other Network Connectivity Devices 88TCP/IP Protocols 89The TCP/IP Architecture 90

Introduction 112Overview 112

Distributed Application Architecture 113Traditional Client-Server Systems 113Thin- versus Thick-Client Systems 113Web-Based Client-Server Systems 114

Application Service Components 117Third-Party Components 119Integrated Application Components 119Dynamic Link Library (DLL) 119Potential DLL-Related Errors 122Scripts 123

Web Application Component Architecture 123

Server-Side Components 123Core Application Service Components 124Markup Language Pages 125

Web-to-Database Connectivity 125Other Application Service Components 128Client-Side Components 130

Testing Considerations 141

Script Testing Issues 143Characteristics of a Script 143Use of Scripts in Web Applications 144Testing Scripts in Web Applications 145Coding-Related Problems 145Script Configuration Testing 147

Bibliography 147

Introduction 150What Is a Mobile Web Application? 150Various Types of Mobile Web Client 151

Palm-Sized PDA Devices 151

Smart Phones or Mobile Phone/PDA Combos 157

Mobile Web Application Platform

Microbrowsers 159Web Clipping Application: How Does It Work? 161Handheld Device Hardware Restrictions 163Software-Related Issues 164Wireless Network Issues 166Wireless Network Standards 166

Wireless LAN and Bluetooth 170Other Software Development Platforms

and Support Infrastructures 171

The Device Technology Converging Game:

Bibliography and Additional Resources 172

Bibliography 172

LogiGear One-Page Test Plan 184

Developing a One-Page Test Plan 185Step 1: Test Task Definition 185Step 2: Task Completion Time 185Step 3: Placing the Test Task into Context 186Step 4: Table Completion 186Step 5: Resource Estimation 186Using the LogiGear One-Page Test Plan 187

Introduction 194

Functionality of the Sample Application 196

Installing the Sample Application 197

Introduction 204

Step 1: Testing-Task Definitions for the Sample Application 205Step 2: Task Completion Time 205Step 3: Placing Test Tasks into the Project Plan 209Step 4: Calculate Hours and Resource Estimates 210

Bibliography 212

Part Three Testing Practice 213

Introduction 216User Interface Design Testing 216

Profiling the Target User 217

Application-Specific Experience 218Considering the Design 220

User Interaction (Data Input) 225Data Presentation (Data Output) 240

User Interface Implementation Testing 243

Miscellaneous User Interface Elements 243Display Compatibility Matrix 246

Usability and Accessibility Testing 247

Introduction 254

An Example of Cataloging Features

in Preparation for Functional Tests 254

Testing the Sample Application 254

Introduction 270Common Server-Side Testing Issues 271

Resource Issues 274Backup and Restore Issues 275

Multithreading Issues 277

Using Monitoring Tools 284Creating Test Interfaces or Test Drivers 289The Testing Environment 291Working with Live Systems 292Resetting the Server 292Using Scripts in Server-Side Testing 293

Bibliography 294

Testing Tools for Run-Time Testing 295

Introduction 298

Batch Files and Shell Scripts 301

Why Not Just Use a Compiled Program Language? 302What Should You Script? 303

Application of Scripting to Testing Tasks 303

System Administration: Automating Tasks 303Discovering Information about the System 304Testing the Server Directly: Making Server-Side Requests 305Working with the Application Independent of the UI 306Examining Data: Log Files and Reports 307Using Scripts to Understand Test Results 308Using Scripts to Improve Productivity 309

A Script to Test Many Files 309

A Set of Scripts That Run Many Times 310Executing Tests That Cannot Be Run Manually 311

Scripting Project Good Practice 311

Where to Find Tools and Download Scripts 316

Bibliography and Useful Reading 316

Chapter 14 Database Tests 317

Introduction 318

Structured Query Language 320Database Producers and Standards 321

Client/SQL Interfacing 325

Microsoft Approach to CLI 325

Database Testing Considerations 349Bibliography and Additional Resources 350

Bibliography 350

Introduction 354

Types of Help Systems 354Application Help Systems 354Reference Help Systems 355Tutorial Help Systems 355Sales and Marketing Help Systems 355Evaluating the Target User 355Evaluating the Design Approach 356Evaluating the Technologies 356Standard HTML (W3 Standard) 356

Interaction between the Application and the Help System 361

Bibliography 366

Introduction 368The Roles of Installation/Uninstallation Programs 369

Installer 369Uninstaller 371

Installation Sources and Destinations 373Server Distribution Configurations 373Server-Side Installation Example 374

Bibliography and Additional Resources 394

Bibliography 394

Introduction 396

Approaching Configuration

Considering Target Users 400When to Run Compatibility and Configuration Testing 400Potential Outsourcing 401

Comparing Configuration Testing

Configuration/Compatibility Testing Issues 403

COTS Products versus Hosted Systems 403Distributed Server Configurations 404

Strategies, People, and Processes 425Education 425Corporate Security Policies 426

Authentication and Authorization 427Passwords 427Authentication between Software Applications

Cryptography 428Other Web Security Technologies 430Perimeter-Based Security: Firewalls, DMZs,

and Intrusion Detection Systems 432Firewalls 432

Intrusion Detection Systems (IDS) 435

Common Vulnerabilities and Attacks 435

Software Bugs, Poor Design, and Programming Practice 436

Malicious Input Data 439Command-Line (Shell) Execution 439Backdoors 440JavaScript 440

Java 440ActiveX 441Cookies 441Spoofing 442

Testing Goals and Responsibilities 446

Functionality Side Effect: An Error-Handling Bug Example 446

Testing the Requirements and Design 449Requirements Are Key 449Trusted Computational Base (TCB) 450

Which Resources Need to Be Protected? 451Client Privacy Issues: What Information Needs to Be Private? 451Testing the Application Code 452Backdoors 452Exception Handling and Failure Notification 452

ID and Password Testing 453Testing for Information Leaks 453Random Numbers versus Unique Numbers 454Testing the Use of GET and POST 454Parameter-Tampering Attacks 455SQL Injection Attacks 456

Testing for Buffer Overflows 458Testing for Bad Data 459Reliance on Client-Side Scripting 460When Input Becomes Output 460Testing Third-Party Code 461Known Vulnerabilities 461

Penetration Testing 463Testing with User Protection via Browser Settings 465Testing with Firewalls 468The Challenges Testers Face 471

Bibliography and Additional Resources 476

Bibliography 476

Tools 478

Introduction 480

Determining Acceptable Response Time

or Acceptable User Experience 481Response Time Definition 482Performance and Load Stress Testing Definitions 483Searching for Answers 484

Performance Testing Key Factors 487

Workload 489System Environment and Available Resources 489

Key Factors Affecting Response Time or Performance 492

Three Phases of Performance Testing 493Setting Goals and Expectations

What Are You Up Against? 496What If Written Requirements Don’t Exist? 496

Problems Concerning Workloads 504Selecting Performance Metrics 505

Throughput Calculation Example 506

Which Tests to Run and When to Start 508Tool Options and Generating Loads 512

Analyzing and Reporting Collected Data 513

Identifying Baseline Configuration and Performance Requirements 515Determining the Workload 515Determining When to Begin Testing 515

Determine Whether the Testing Process Will Be Hardware-Intensive or Software-Intensive 516Developing Test Cases 516

Setting Up the Test Bed 517Setting Up the Test Suite Parameters 518Performance Test Run Example 518

Bibliography 525

Introduction 528Testing Mobile versus Desktop Web Applications 528

Add-on Installation Tests 536Data Synchronization-Related Tests 536

UI Implementation and Limited Usability Tests 537

UI Guideline References 538Browser-Specific Tests 539Platform-Specific Tests 539Platform or Logo Compliance Tests 540Configuration and Compatibility Tests 540

Testing Web Applications Using

an Emulation Environment 544Testing Web Applications Using

the Physical Environment 545

Survey of Mobile Testing Support Tools 546

Device and Browser Emulators 546

OpenWave 547Nokia 548YoSpace 548Microsoft 548Web-Based Mobile Phone Emulators

Other Testing Considerations 549Bibliography and Additional Resources 550

Bibliography 550

and Regression Testing Tools 559Runtime Error Detectors 561Sample List of Runtime Error-Detection Tools 561Sample List of Web Security Testing Tools 562Java-Specific Testing Tools 564Other Types of Useful Tools 564Database Testing Tools 564Defect Management Tool Vendors 565QACity.Com Comprehensive List of DEFECT TRACKING

Development and Testing Tool Mail-Order Catalogs 566

Introduction 568Textbooks 568

Appendix A LogiGear Test Plan Template 579

Appendix D UI Test-Case Design Guideline: Common Keyboard

Appendix F Web Test-Case Design Guideline: Input Boundary

Testing Applications on the Web introduces the essential technologies, testing

concepts, and techniques that are associated with browser-based applications

It offers advice pertaining to the testing of business-to-business applications,business-to-end-user applications, Web portals, and other Internet-based appli-cations The primary audience is software testers, software quality engineers,quality assurance staff, test managers, project managers, IT managers, busi-ness and system analysts, and anyone who has the responsibility of planningand managing Web-application test projects

Testing Applications on the Web begins with an introduction to the

client-server and Web system architectures It offers an in-depth exploration of Webapplication technologies such as network protocols, component-based archi-tectures, and multiple server types from the testing perspective It then coverstesting practices in the context of various test types from user interface tests toperformance, load, and stress tests, and security tests Chapters 1 and 2 present

an overview of Web testing Chapters 3 through 6 cover methodology andtechnology basics, including a review of software testing basics, a discussion

on networking, an introduction to component-based testing, and an overview

of the mobile device platform Chapters 7 through 9 discuss testing planningfundamentals, a sample application to be used as an application under test(AUT) throughout the book, and a sample test plan Chapters 10 through 20discuss test types that can be applied to Web testing Finally, Chapters 21 and

22 offer a survey of Web testing tools and suggest where to go for additionalinformation

Testing Applications on the Web answers testing questions such as, “How do

networking hardware and software affect applications under test?” “What areWeb application components, and how do they affect my testing strategies?”

Preface

xxi

“What is the role of a back-end database, and how do I test for related errors?” “How do I test server-side software?” “What are performance,stress, and load tests, and how do I plan for and execute them?” “What do Ineed to know about security testing, and what are my testing responsibili-ties?” “What do I need to consider in testing mobile Web applications?”With a combination of general testing methodologies and the informationcontained in this book, you will have the foundation required to achieve thesetesting goals—maximizing productivity and minimizing quality risks in aWeb application environment.

database-Testing Applications on the Web assumes that you already have a basic

under-standing of software testing methodologies, including test planning, test-casedesign, and bug report writing Web applications are complex systems thatinvolve numerous components: servers, browsers, third-party software andhardware, protocols, connectivity, and much more This book enables you toapply your existing testing skills to the testing of Web applications

N OT E This book is not an introduction to software testing If you are looking for fundamental software testing practices, you will be better served by reading

Testing Computer Software,Second Edition, by Kaner, Cem, Jack Falk, and Hung Q Nguyen (Wiley, 1999) For additional information on Web testing and other testing techniques and resources, visit www.QAcity.com.

We have enjoyed writing this book and teaching the Web application testingtechniques that we use every day to test Web-based systems We hope that youwill find here the information you need to plan for and execute a successfultesting strategy that enables you to deliver high-quality applications in anincreasingly distributed-computing, market-driven, and time-constrainedenvironment in this era of new technology

Writing about Web testing is challenging because the field involves the dependence of so many different technologies and systems It’s not enough

inter-to write about the client Certainly, the client software is the part of the cation that is the most visible to the customer, and it’s the easiest to write about(authors can just repackage the same old stuff published about applications

appli-in general Hung, Michael, and Bob do provide client-side guidance, but theirgoal is to provide information that is specific to Web applications (For more

generic material, you can read Testing Computer Software, Second Edition,

Wiley, 1999.)But client-side software is just the tip of the iceberg The application dis-plays itself to the end user as the client, but it does most of its work in con-junction with other software on the server-side, much of it written andmaintained by third parties For example, the application probably stores andretrieves data via third-party databases If it sells products or services, it prob-ably clears customer orders with the customer’s credit card company It mightalso check its distributor for available inventory and its shippers for the cost ofshipping the software to the customer The Web application communicateswith these third parties through network connections written by third parties.Even the user interface is only partially under the application developer’s control—the customer supplies the presentation layer: the browser, the musicand video player, and perhaps various other multimedia plug-ins

The Web application runs on a broader collection of hardware and softwareplatforms than any other type of application in history Attributes of these plat-forms can change at any time, entirely outside of the knowledge or control ofthe Web application developer

Foreword

xxiii

In Testing Applications on the Web, Nguyen, Hackett, and Johnson take this

complexity seriously In their view, a competent Web application tester mustlearn the technical details of the systems with which the application under testinteracts To facilitate this, they survey many of those systems, explaining howapplications interact with them and providing testing tips

As a by-product of helping testers appreciate the complexity of the Web

test-ing problem, the first edition of Testtest-ing Applications on the Web became the first

book on gray-box testing In so-called black-box testing, we treat the softwareunder test as a black box We specify the inputs, we look at the outputs, but wecan’t see inside the box to see how it works The black-box tester operates at

the customer’s level, basing tests on knowledge of how the system should

work In contrast, the white-box tester knows the internals of the software, anddesigns tests with direct reference to the program’s source code The gray-boxtester doesn’t have access to the source code, but he or she knows much moreabout the underlying architecture and the nature of the interfaces between theapplication under test and the other software and the operating systems.The second edition continues the gray-box analysis by deepening the dis-cussions in the first edition It also adds several new chapters to address business-critical testing issues from server-side, performance- and application-level security testing to the latest mobile Web application testing A finalstrength of the book is the power of the real-world example Hung QuocNguyen is the president of the company that published TRACKGEAR, a Web-based bug tracking system, enabling the authors can give us the inside story ofits development and testing

This combination of a thorough and original presentation of a style of sis, mixed with detailed insider knowledge is a real treat to read It teaches usabout thinking through the issues involved when the software under testinteracts in complex ways with many other programs, and it gives the book avalue that will last well beyond the specifics of the technologies describedtherein

analy-Cem Kaner, J.D., Ph D

Professor of Computer SciencesFlorida Institute of Technology

While it is our names that appear on the cover, over the years, many peoplehave helped with the development of this book We want to particularly thankBrian Lawrence, for his dedication in providing thorough reviews and criticalfeedback We all thank Cem Kaner for his guidance, friendship, and generosity,and for being there when we needed him We thank, too, Jesse Watkins-Gibbsfor his work on examples and sample code, as well as for his technical exper-tise and his commitment to getting our book done

We would also like to thank our professional friends who took time outfrom their demanding jobs and lives to review and add comment on the book:Yannick Bertolus, George Hamblin, Elisabeth Hendrickson, NematolahKashanian, Pat McGee, Alberto Savoia, and Garrin Wong We would like tothank our copyeditor Janice Borzendowski We also want to thank the follow-ing people for their contributions (listed in alphabetical order): James L Carr,William Coleman, Norm Hardy, Pam Hardy, Thomas Heinz, Chris Hibbert,Heather Ho, Brian Jones, Denny Nguyen, Kevin Nguyen, Wendy Nguyen,Steve Schuster, Kurt Thams, Anne Tran, Dean Tribble, and Joe Vallejo Finally,

we would like to thank our colleagues, students, and staff at LogiGear ration, for their discussions and evaluations of the Web testing training mate-rial, which made its way into this book And thanks to our agent ClaudetteMoore of Moore Literacy Agency

Corpo-Certainly, any remaining errors in the book are ours

Acknowledgments

xxv

Hung Q Nguyenis Founder, President, and CEO of LogiGear Corporation.Nguyen has held leadership roles in business management, product develop-ment, business development, engineering, quality assurance, software testing,and information technology Hung is an international speaker and a regularcontributor to industry publications He is the original architect of TRACK-GEAR, a Web-based defect management system Hung also teaches softwaretesting for the University of California at Berkeley and Santa Cruz Extension,and LogiGear University.

Hung is the author of Testing Applications on the Web, First Edition (Wiley 2000); and with Cem Kaner and Jack Falk, he wrote the best-selling book Test-

ing Computer Software (ITP/Wiley 1993/1999) He holds a Bachelor of Science

in Quality Assurance from Cogswell Polytechnical College, and is an Certified Quality Engineer and a member of the Advisory Council for theDepartment of Applied Computing and Information Systems at UC BerkeleyExtension

ASQ-You can reach Hung at hungn@logigear.com; or, to obtain more informationabout LogiGear Corporation and Hung’s work, visit www.logigear.com

Bob Johnson has been a software developer, tester, and manager of bothdevelopment and testing organizations With over 20 years of experience insoftware engineering, Bob has acquired key strengths in building applications

on a variety of platforms Bob’s career in software development ranges fromWeb programming to consulting on legal aspects of e-commerce to the require-ment and review process Whether working in test automation, Web security, orback-end server testing, Bob is at the forefront of emerging technologies

About the Authors

xxvii

In addition to participating in the Los Altos Workshops on Software Testing

(LAWST), Bob has written articles for IEEE Software, Journal of Electronic

Commerce, and Software Testing and Quality Engineering He can be reached at

rjohnson@testingbook.com

Michael Hackett isVice President and a founding partner of LogiGear ration He has over a decade of experience in software engineering and thetesting of shrink-wrap and Internet-based applications Michael has helpedwell-known companies release applications ranging from business productiv-ity to educational multimedia titles, in English as well as a multitude of otherlanguages Michael has taught software testing for the University of California

Corpo-at Berkeley Extension, the Software Productivity Center in Vancouver, theHong Kong Productivity Centre, and LogiGear University Michael holds aBachelor of Science in Engineering from Carnegie-Mellon University He can

be reached at michaelh@logigear.com

PA R T

One Introduction

C H A P T E R

1

Why Read This Chapter?

The goal of this book is to help you effectively plan for and conduct the testing

of Web-based applications developed for fixed clients, systems in fixed tions such as desktop computers, as well as for mobile clients such as mobilephones, PDAs (Personal Digital Assistants), and portable computers Thisbook will be more helpful to you if you understand the philosophy behind itsdesign

loca-Software testing practices have been improving steadily over the past fewdecades Yet, as testers, we still face many of the same challenges that we havefaced for years We are challenged by rapidly evolving technologies and theneed to improve testing techniques We are also challenged by the lack ofresearch on how to test for and analyze software errors from their behavior, asopposed to at the source code level We are challenged by the lack of technicalinformation and training programs geared toward serving the growing popu-lation of the not-yet-well-defined software testing profession Finally, we arechallenged by limited executive management support as a result of manage-ment’s underestimation and lack of attention to the cost of quality Yet, in

Welcome to Web

Testing

C H A P T E R

This chapter offers a historical perspective on the changing objectives of ware testing It touches on the gray-box testing approach and suggests theimportance of having a balance of product design, both from the designer’sand the user’s perspective, and system-specific technical knowledge It alsoexplores the value of problem analysis to determine what to test, when to test,and where to test Finally, this chapter will discuss what assumptions this bookhas about the reader

soft-The Evolution of Software Testing

As the complexities of software development have evolved over the years, thedemands placed on software engineering, information technology (IT), andsoftware quality professionals have grown and taken on greater relevance Weare expected to check whether the software performs in accordance with itsintended design and to uncover potential problems that might not have beenanticipated in the design We are expected to develop and execute more tests,faster, and more often Test groups are expected to offer continuous assess-ment on the current state of the projects under development At any givenmoment, we must be prepared to report explicit details of testing coverage andstatus, the health or stability of the current release, and all unresolved errors.Beyond that, testers are expected to act as user advocates This often involves

TOPICS COVERED IN THIS CHAPTER

◆ Introduction

◆ The Evolution of Software Testing

◆ The Gray-Box Testing Approach

◆ Real-World Software Testing

◆ Themes of This Book

◆ What’s New in the Second Edition

today’s world of Internet time, the systems under test are getting more complex

by the day, and resources and testing time are in short supply The quicker wecan get the information that we need, the more productive and more success-ful we will be at doing our job The goal of this book is to help you do your jobeffectively

anticipating usability problems early in the development process so thoseproblems can be addressed in a timely manner.

In the early years, on mainframe systems, many users were connected to acentral system Bug fixing involved patching or updating the centrally storedprogram This single fix would serve the needs of hundreds or thousands ofindividuals who used the system

As computing became more decentralized, minicomputers and puters were run as stand-alone systems or on smaller networks There weremany independent computers or local area networks, and a patch to the code

microcom-on microcom-one of these computers updated relatively fewer people Mass-market ware companies sometimes spent over a million dollars sending disks to reg-istered customers just to fix a serious defect Additionally, technical supportcosts skyrocketed

soft-As the market has broadened, more people use computers for more thingsand rely more heavily on computers, hence the consequences of softwaredefects has risen every year It is impossible to find all possible problems bytesting, but as the cost of failure has gone up, it has become essential to do risk-based testing In a risk-based approach, you ask questions like these:

■■ Which areas of the product are so significant to the customer or soprone to serious failure that they must be tested with extreme care?

■■ For the average area, and for the program as a whole, how much testing

is enough?

■■ What are the risks involved in leaving a certain bug unresolved?

■■ Are certain components so unimportant as to not merit testing?

■■ At what point can a product be considered adequately tested and readyfor market?

■■ How much longer can the product be delayed for testing and fixingbugs before the market viability diminishes the return on investment?

Tracking bugs, analyzing and assessing their significance are priorities.Management teams expect development and IT teams, testing and qualityassurance staff, to provide quantitative data regarding test coverage, the status

of unresolved defects, and the potential impact of deferring certain defects Tomeet these needs, testers must understand the products and technologies theytest They need models to communicate assessments of how much testing hasbeen done in a given product, how deep testing will go, and at what point theproduct will be considered adequately tested Given better understanding oftesting information, we make better predictions about quality risks

In the era of the Internet, the connectivity that was lost when computingmoved from the mainframe model to the personal computer (PC) model,

in effect, has been reestablished Personal computers are effectively networkedover the Internet Bug fixes and updated builds are made available—sometimes on a daily basis—for immediate download over the Internet.Product features that are not ready by ship date are made available later in

service packs The ability to distribute software over the Internet has brought

down much of the cost that is associated with distributing some applicationsand their subsequent bug fixes

Although the Internet offers connectivity for PCs, it does not offer the trol over the client environment that was available in the mainframe model.The development and testing challenges with the Graphical User Interface(GUI) and event-based processing of the PC are enormous because the clientsattempt remarkably complex tasks on operating systems (OSs) as different asUNIX, Macintosh OS, Linux, and the Microsoft OSs They run countless com-binations of processors, peripherals, and application software Additionally,the testing of an enterprise client-server system may require the consideration

con-of thousands con-of combinations con-of OSs, modems, routers, and client-serversoftware components and packages Web applications increase this complex-ity further by introducing browsers and Web servers into the mix Further-more, wireless networks are becoming more pervasive, and their bandwidthscontinue to improve On the client-side, computer engineers continue toadvance in building smaller, yet more powerful portable or mobile devices.Communication and wearable devices and Internet appliances are expandingthe possible combinations of Web client environments beyond the desktopenvironments On the server-side, software components that were normallylocated in a company’s enterprise server are making their move toward theapplication services or Web services model In this model, the components will

be hosted outside of the corporate enterprise server, usually at the third-partyASPs (application service providers), adding more challenges to the testing ofInternet-based systems

Software testing plays a more prominent role in the software developmentprocess than ever before Developers are paying more attention to buildingtestability into their code, as well as coming up with more ways to improve theunit-test framework around their production code Companies are allocatingmore money and resources for testing because they understand that their rep-utations and success rest on the quality of their products, or that their failure ismore probable due to poor product and service quality The competitiveness ofthe computing industry (not to mention the savvy of most computer users) haseliminated most tolerance for buggy software Nevertheless, many companies

believe that the only way to compete in Internet time is to develop software as

rapidly as possible Short-term competitive issues often outweigh qualityissues One consequence of today’s accelerated development schedules is the

industry’s tendency to push software out into the marketplace as early as sible Development teams get less and less time to design, code, test, andundertake process improvements Market constraints and short developmentcycles often do not allow time for reflection on past experience and considera-tion of more efficient ways to produce and test software.

pos-The Gray-Box Testing Approach

Black-box testing focuses on software’s external attributes and behavior Suchtesting looks at an application’s expected behavior from the user’s point ofview White-box testing (also known as glass-box testing), on the other end ofthe spectrum, tests software with knowledge of internal data structures, phys-ical logic flow, and architecture at the source code level White-box testinglooks at testing from the developer’s point of view Both black-box and white-box testing are critically important complements of a complete testing effort.Individually, they do not allow for balanced testing Black-box testing can beless effective at uncovering certain error types, such as data-flow errors orboundary condition errors at the source level White-box testing does not read-ily highlight macro-level quality risks in operating environment, compatibil-ity, time-related errors, and usability

Gray-box testing incorporates elements of both black-box and white-boxtesting It considers the outcome on the user end, system-specific technicalknowledge, and operating environment It evaluates application design in the

context of the interoperability of system components The gray-box testing

approach is integral to the effective testing of Web applications because Webapplications comprise numerous components, both software and hardware.These components must be tested in the context of system design to evaluatetheir functionality and compatibility

In our view, gray-box testing consists of methods and tools derived from theknowledge of the application internals and the environment with which itinteracts The knowledge of the designer’s intended logic can be applied intest design and bug analysis to improve the probability of finding and repro-ducing bugs See Chapter 5, “Web Application Components,” the section enti-tled “Testing Discussion,” for an example of gray-box testing methods

Here are several unofficial definitions for gray-box testing from the LosAltos Workshop on Software Testing (LAWST) IX (For more information onLAWST, visit www.kaner.com.)

Gray-box testing—Using inferred or incomplete structural or design mation to expand or focus black-box testing —Dick Bender

infor-Gray-box testing—Tests designed based on the knowledge of algorithms,internal states, architectures, or other high-level descriptions of programbehavior —Doug Hoffman

Gray-box testing—Tests involving inputs and outputs; but test design iseducated by information about the code or the program operation of akind that would normally be out of scope of the view of the tester —Cem Kaner

Gray-box testing is well suited for Web application testing because it factors

in high-level design, environment, and interoperability conditions It willreveal problems that are not as easily considered by a black-box or white-boxanalysis, especially problems of end-to-end information flow and distributedhardware/software system configuration and compatibility Context-specificerrors that are germane to Web systems are commonly uncovered in thisprocess

Another point to consider is that many of the types of errors that we run into

in Web applications might well be discovered by black-box testers, if only wehad a better model of the types of failures for which to look and design tests.Unfortunately, we are still developing a better understanding of the risks thatare associated with the new application and communication architectures

Therefore, the wisdom of traditional books on testing (e.g., Testing Computer

Software, 2nd ed., John Wiley & Sons, Inc (1999), by Kaner, Falk, and Nguyen

and Lessons Learned in Software Testing, John Wiley & Sons, Inc (2002), by

Kaner, Bach, and Pettichord) will not fully prepare the black-box tester tosearch for these types of errors If we are equipped with a better under-standing of the system as a whole, we’ll have an advantage in exploring thesystem for errors and in recognizing new problems or new variations of olderproblems

As testers, we get ideas for test cases from a wide range of knowledge areas.This is partially because testing is much more effective when we know andmodel after the types of bugs we are looking for We develop ideas of whatmight fail, and of how to find and recognize such a failure, from knowledge ofmany types of things [e.g., knowledge of the application and system architec-ture, the requirements and use of this type of application (domain expertise),and software development and integration] As testers of complex systems, weshould strive to attain a broad balance in our knowledge, learning enoughabout many aspects of the software and systems being tested to create a bat-tery of tests that can challenge the software as deeply as it will be challenged

in the rough and tumble of day-to-day use

Finally, we are not suggesting that every tester in a group be a gray-box

tester We have seen a high level of success in several test teams that have a mix

of different types of testers, with different skill sets (e.g., subject matter expert,

database expert, security expert, API testing expert, test automation expert,etc.) The key is, within that mix, for at least some of the testers to understandthe system as a collection of components that can fail in their interaction witheach other, and these individuals must understand how to control and how tosee those interactions in the testing and production environments.

Real-World Software Testing

Web businesses have the potential to be high-profit ventures In the dot-comera, venture capitalists could support a number of losing companies as long asthey had a few winners to make up for their losses A CEO has three to fiveyears to get a start-up ready for IPO (six months to prove that the prototypeworks, one or two years to generate some revenue—hence, justifying the busi-ness model—and the remainder of the time to show that the business can beprofitable someday) It is always a challenge to find enough time and quali-fied personnel to develop and deliver quality products in such a fast-pacedenvironment

Although standard software development methodologies such as ity Maturity Model (CMM) and ISO-9000 have been available, they are not yetwell accepted by aggressive start-up companies These standards and meth-ods are great practices, but the fact remains that many companies will rely onthe efforts of a skilled development and testing staff, rather than a process thatthey fear might slow them down In that situation, no amount of improvedstandards and process efficiencies can make up for the efforts of a skilleddevelopment and testing staff That is, given the time and resource constraints,they still need to figure out how to produce quality software

Capabil-The main challenge that we face in Web application testing is to learn theassociated technologies in order to have a better command over the environ-ment We need to know how Web technologies affect the interoperability ofsoftware components, as well as Web systems as a whole Testers also need toknow how to approach the testing of Web-based applications This requiresbeing familiar with test types, testing issues, common software errors, and thequality-related risks that are specific to Web applications We need to learn,and we need to learn fast Only with a solid understanding of software testingbasics and a thorough knowledge of Web technologies can we competentlytest Web-based systems

The era of high-profit dot-com ventures has ended Many businesses based

on the dot-com model have closed their doors The NASDAQ Stock Exchangewithin three years of March 2000, from its best performance of above 5,000points, has dropped to as low as 1,100 We have learned many great and

painful business lessons from this era, but unfortunately, not many related lessons However, the good news is, in a difficult economic time, themarket demand is low, hence customers are in control They want to pay lessmoney for more, and demand higher-quality products and services Bugs left

quality-in the product could mean anythquality-ing from high support costs to loss of sales

to contract cancellation That is money loss that companies can normallyabsorb during the good economic times, but not in the bad times We haveseen many movements and initiatives in which executive staffs are payingmore attention and asking more interesting quality-related questions They areasking questions like, “How can we produce better software (higher-quality),faster (improve time-to-market) at a lower cost (lower production and qualitycosts)?” While this type of question has been asked before, executive staffsnow have more time to listen and are more willing to support the QA effort Italso means a lot more demand is placed on testing, which requires QA staff to

be more skilled in testing strategy, better educated in software engineering andtechnology, better equipped with testing practices, and savvier in the conceptand application of software testing tools

Themes of This Book

The objective of this book is to introduce testers into the discipline of gray-boxtesting, by offering readers information about the interplay of Web applica-tions, component architectural designs, and their network systems We expectthat this will help testers develop new testing ideas, enabling them to uncoverand troubleshoot new types of errors and conduct more effective root-causeanalyses of software failures discovered during testing or product use Thediscussions in this book focus on determining what to test, where to test, andwhen to test As appropriate, real-world testing experiences and examples oferrors are included

To effectively plan and execute the testing of your Web application, youneed to possess the following qualities: good software testing skill; knowledge

of your application, which you will need to provide; knowledge of Web nologies; understanding of the types of tests and their applicability to Webapplication; knowledge of several types of Web application-specific errors (soyou know what to look for); and knowledge of some of the available tools andtheir applicability, which this book offers you (See Figure 1.1.)

tech-Based on this knowledge and skill set, you can analyze the testing ments to come up with an effective plan for your test execution If this is whatyou are looking for, this book is for you It is assumed that you have a solidgrasp of standard software testing practices and procedures