Software engineering is the only engineering discipline where product testing is amajor technical and organizational concern, as well as an important cost factor.Several factors contribu
TEST DATA GENERATION 141
8.1 Test Generation and Target Attributes / 143 8.2 Test Outcomes / 146
8.3 Test Generation Requirements / 148 8.4 Test Generation Criteria / 152 8.5 Empirical Adequacy Assessment / 155 8.6 Chapter Summary / 160
8.7 Exercises / 161 8.8 Bibliographic Notes / 162 8.9 Appendix: Mutation Program / 163
9.1 Domain Partitioning / 165 9.2 Test Data Generation from Tabular Expressions / 171 9.3 Test Generation for State Based Systems / 176 9.4 Random Test Data Generation / 184
9.5 Tourism as a Metaphor for Test Data Selection / 188 9.6 Chapter Summary / 190
10.1 Paths and Path Conditions / 194 10.1.1 Execution Paths / 194 10.1.2 Path Functions / 196 10.1.3 Path Conditions / 201 10.2 Control Flow Coverage / 202
10.2.1 Statement Coverage / 202 10.2.2 Branch Coverage / 204 10.2.3 Condition Coverage / 207 10.2.4 Path Coverage / 209 10.3 Data Flow Coverage / 214
10.3.1 Definitions and Uses / 214 10.3.2 Test Generation Criteria / 217 10.3.3 A Hierarchy of Criteria / 220 10.4 Fault-Based Test Generation / 220
10.4.1 Sensitizing Faults / 221 10.4.2 Selecting Input Data for Fault Sensitization / 225 10.4.3 Selecting Input Data for Error Propagation / 227 10.5 Chapter Summary / 228 x CONTENTS
TEST DEPLOYMENT AND ANALYSIS 233
11.1 Dilemmas of Oracle Design / 235 11.2 From Specifications to Oracles / 238 11.3 Oracles for State-Based Products / 242
11.3.1 From Axioms to Oracles / 243 11.3.2 From Rules to Oracles / 244 11.4 Chapter Summary / 250
12.1 Selecting a Specification / 253 12.2 Selecting a Process / 255 12.3 Selecting a Specification Model / 257
12.3.1 Random Test Generation / 257 12.3.2 Pre-Generated Test Data / 263 12.3.3 Faults and Fault Detection / 266 12.4 Testing by Symbolic Execution / 269 12.5 Chapter Summary / 274
13.1 Logical Claims / 281 13.1.1 Concrete Testing / 281 13.1.2 Symbolic Testing / 282 13.1.3 Concolic Testing / 283 13.2 Stochastic Claims: Fault Density / 284 13.3 Stochastic Claims: Failure Probability / 287
13.3.1 Faults Are Not Created Equal / 287 13.3.2 Defining/Quantifying Reliability / 289 13.3.3 Modeling Software Reliability / 291 13.3.4 Certification Testing / 294
13.3.5 Reliability Estimation and Reliability Improvement / 295 13.3.6 Reliability Standards / 299
13.3.7 Reliability as an Economic Function / 300 13.4 Chapter Summary / 307 xi
MANAGEMENT OF SOFTWARE TESTING 311
T CHIER A M ILI
In this part we introduce software testing by discussing what makes software engineer- ing so special that testing should occupy such an important part of its lifecycle Then we survey software qualities that testing techniques may be used to assess Finally we review the various lifecycle models of software testing that may be followed depending on the context and goal of testing.
Software Testing: Concepts and Operations, First Edition Ali Mili and Fairouz Tchier. © 2015 John Wiley & Sons, Inc Published 2015 by John Wiley & Sons, Inc.
On the face of it, software engineering sounds like an engineering discipline among others, such as chemical engineering, mechanical engineering, civil engineering, and electrical engineering We will explore, in this chapter, in what way and to what extent software engineering differs from other engineering disciplines.
Civil engineering and mechanical engineering date back to antiquity or before, as one can see from various sites (buildings, road networks, utility infrastructures, etc.) around the Mediterranean basin Chemical engineering (Lavoisier and others) and electrical engineering (Franklin and others) can be traced back to the eighteenth century Nuclear engineering (Pierre and Marie Curie) emerged at the turn of the twentieth century and industrial engineering emerged around the time of the Second World War, with issues of logistics By contrast, software engineering is a comparatively young discipline, emerging as it did in the second half of the twentieth century The brief history of this discipline can be divided into five broad eras, lasting approximately one decade each, which are as follows:
• The Sixties: The Era of Pioneers.This era marks the first time that practitioners and researchers came face to face with the complexities, paradoxes, and anomalies of software engineering Software projects of this era were ventures into unchar- tered territory, characterized by high levels of risk, unpredictable outcomes, and massive cost and schedule overruns The programming languages that were dominant in this era are assembler, Fortran, Cobol, and (in academia) Algol.
Software Testing: Concepts and Operations, First Edition Ali Mili and Fairouz Tchier. © 2015 John Wiley & Sons, Inc Published 2015 by John Wiley & Sons, Inc.
• The Seventies: Structured Software Engineering.This era is characterized by the general belief that software engineering problems are of a technical nature and that if we evolved techniques for software specification, design, and verification to control complexity, all software engineering problems would be resolved.
Given that structure is our main intellectual tool for dealing with complexity, this era has seen the emergence of a wide range of structured techniques, including structured programming, structured design, structured analysis, structured speci- fications, etc The programming languages that were dominant in this era are C and (in academia) Pascal.
• The Eighties: Knowledge-Based Software Engineering.This era is characterized by the realization that software engineering problems are of a managerial and organizational nature more than a technical nature This realization was concur- rent with the emergence of the Fifth Generation Computing initiative, which started in Japan and spread across the globe (the United States, Europe, Canada), and was focused on thinking machines designed with extensive use of artificial intelligence techniques This general approach permeated the discipline of soft- ware engineering with the emergence of knowledge-based software engineering techniques The programming languages that were dominant in this era are Prolog, Scheme/Lisp, and Ada.
• The Nineties: Reuse-Based Software Engineering As it became increasingly clear that fifth-generation computing was not delivering on its promise, and worldwide fifth-generation initiatives were fading, software researchers and practitioners turned their attention to reuse as a possible savior of the discipline.
Software engineering is, after all, the only discipline where reuse is not an inte- gral part of the routine engineering process It was felt that if only software engi- neers had large databases of reusable software components readily available, the industry would achieve great gains in productivity, quality, time to market, and reduced process risk This evolution was concurrent with the emergence of object-oriented programming, which supports a bottom–up design discipline that facilitates product reuse The programming languages that were dominant in this era are C, C++, Eiffel, and Smalltalk.
• The First Decade of the Millennium: Lightweight Software Engineering.While software reuse is not practical as a general paradigm in software engineering, it is feasible in limited application domains, giving rise toproduct line engineer- ing Other attributes of this era include Java programming, with its focus on web applications; agile programming, with its focus on rapid and flexible response to change; and component-based software engineering, with its focus on software architecture and software composition The programming languages that were dominant in this era are Java, C++, and (in academia) Python.
Perhaps as result of this young and eventful history, the discipline of software engi- neering is characterized by a number of paradoxes and counter-intuitive properties, which we explore in this chapter.
Nowadays, software runs all aspects of modern life and accounts for a large and increasing share of the world economy This trend started slowly with the advent of computing in the middle of the twentieth century and was further precipitated by the emergence of the World Wide Web at the end of the twentieth and the begin- ning of the twenty-first century This phenomenon has spawned a great demand for software products and services and generated a market pressure that the software industry takes great pains to cater to.
Many fields of science and engineering (such as bioinformatics, medical informat- ics, weather forecasting, and modeling and simulation) are so dependent on software that they can almost be considered as mere applications of software engineering Also, it is possible to observe that many computer science curricula are slowly inching toward more software engineering contents at the expense of traditional theoretical material, which may be perceived as less and less relevant to today’s job market.
Some engineering colleges are preempting the trend by starting software engineering degrees in computer science departments or by starting complete software engineering departments alongside traditional computer science departments.
As software proliferates into diverse and critical domains, its dependability has become paramount The burgeoning demand for software necessitates high product quality, particularly in applications that impact lives, missions, and financial stability Ensuring software's ability to perform its intended function with unwavering reliability is now indispensable for safeguarding crucial operations and sensitive data.
This requires that we deploy a wide range of techniques, including the following:
• Process controls, ensuring that software products are developed and evolved according to certified, mature processes.
• Product controls, ensuring that software products meet quality standards com- mensurate with their application domain requirements; this is achieved by a com- bination of techniques, including static analysis, dynamic testing, reliability estimation, fault tolerance, etc.
m4(a,N); case 5: m5(a,N);}
for (int k=0; k