Luận án tiến sĩ: Exploiting design patterns for improved efficiency in the testing of object-oriented software

In short, a software design pattern is a description of communicating classes and objects that are customized to solve a general design problem in a specific context.. An analogy to the

Exploiting Design Patterns for Improved Efficiency

In the Testing of Object-Oriented Software

University of South Carolina, 2003

Submitted in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy in the Department of Computer Science and Engineering College of Engineering and Information Technology

University of South Carolina

UMI Number: 3245380

INFORMATION TO USERS

The quality of this reproduction is dependent upon the quality of the copy submitted Broken or indistinct print, colored or poor quality illustrations and photographs, print bleed-through, substandard margins, and improper alignment can adversely affect reproduction

In the unlikely event that the author did not send a complete manuscript and there are missing pages, these will be noted Also, if unauthorized copyright material had to be removed, a note will indicate the deletion

®

UMI

UMI Microform 3245380 Copyright 2007 by ProQuest Information and Learning Company

All rights reserved This microform edition is protected against unauthorized copying under Title 17, United States Code

ProQuest Information and Learning Company

300 North Zeeb Road P.O Box 1346 Ann Arbor, MI 48106-1346

Acknowledgements

As my time at Carolina draws to a close, my primary expression of graditude is to

my Lord and Saviour, Jesus Christ, who provided the opportunity to seek this degree and the strength to complete the process

A debt of gratitude is owed to my advisor, Dr John Bowles, for his guidance over

the years I appreciate his patience as he endured many Friday afternoon meetings My thanks also go to the members of my committee for their willingness to serve

Finally, { must thank my family for putting up with my constant travels to

Columbia and the stacks of books scattered throughout the house I’m sure they’re just as happy as I am to see this task accomplished

ii

Abstract

The use of design patterns in software development has become more attractive given the rise in popularity of development tools utilizing a model driven architecture (MDA) philosophy If a given piece of software is designed from a collection of standard design patterns, we should be able to test that software based on the designer’s intent as captured by those patterns

In testing object oriented software, we must be aware that the state of an object at any time depends on the sequence of messages received by the object up to that point in time As these messages are passed by method calls, the number of methods calls and the order in which they occur must be given consideration A metric based on method

coverage is suitable in this context

This dissertation examines the utility of design patterns in the practice of software testing We employ a metric of method coverage to gauge the efficiency obtained by the inclusion of design patterns in two traditional test procedures, statistical design of

experiments and reliability block diagrams In the process, a new type of software system diagram, the Pattern Block Diagram, is developed

ili

Design Pafferns HH» HH HH nh nh nhe nà 11

Software TestInE ch KÝ nhi, 19 Testing Object-oriented Soffwar€ coi 32 Design Patterns and the Statistical Design of Experiments

in the Testing of Object-oriented Software 51 Design Patterns and Reliability Block Diagrams 86

;{2t-)4-14-@ŒHdaaaddidiẳẢẢ 114

iv

List of Tables

1 | TABLE 3.1 — Conditions for Path Traversal in White Box Testing 23

3 | TABLE 3.3 — Comparison of Path Coverage Criteria 26

4 | TABLE 3.4 — Guidelines for Equivalence Classes 27

5 | TABLE 4.1 — Which path to follow if x = 0? 32

6 | TABLE 4.1 — node/edge classifications of Rapps and Weyuker 36

7 | TABLE 4.2 - def/use graph sets of Rapps and Weyuker 37

12 | TABLE 6.1 — Patterns of the Invoice Application 86

16 | TABLE 6.5 — Patterns, classes, and method counts for the Invoice 97

20 | TABLE 6.9 — Driver function method counts for the Invoice Application | 99

21 | TABLE 6.10 — Strategy/Iterator union method counts for the Invoice 99

Application

22 | TABLE 6.12 — Conditional probabilities ( Based on data of Table 6.5 ) 101

23 | TABLE 6.13 — Reduction of the Pattern Block Diagram 106

List of Figures

2 | FIGURE |.2—- A Branch of the Banking Hierarchy 6

5 | FIGURE 1.5 — Class Associations and Cardinalities 9

7 | FIGURE 3.1 — Flowchart for Path Traversal in White Box Testing 21

9 | FIGURE 3.3 — Pseudocode for locating name ‘Smith’ in list 25

10 | FIGURE 4.2 — def/use graph of Rapps and Weyuker 36

13 | FIGURE 4.5 — CoinBox class call graph frame 41

15 | FIGURE 4.7 — State following steps 1,2,3 of Mungara’s algorithm 45

16 | FIGURE 4.8 — Mungara’s completed test case for push( ) method 45

18 | FIGURE 5.2 - Interaction of factors A and C 51

20 | FIGURE 5.4 - Full factorial design for 3 factors 53

22 | FIGURE 5.6 — Factor levels for chemical reaction experiment 55

23 | FIGURE 5.7 — Response table for chemical reaction experiment 55

24 | FIGURE 5.8 — Graphical representation of response table or the chemical | 56

reaction experiment

25 | FIGURE 5.9 — Transaction Processor user interface 57

26 | FIGURE 5.10 — Factor levels for the Transaction Processor 58

28 | FIGURE 5.12 - Response table for 4 factors 59

vi

29 | FIGURE 5.13 - Transaction Processor class count 60

30 | FIGURE 5.14 - Transaction Processor error log 61

31 | FIGURE 5.15 — Completed response table for the Transaction Processor | 62

32 | FIGURE 5.16 - Graphical representation of response table for the 63

Transaction Processor

33 | FIGURE 5.17 - Determining the AB interaction levels 64

34 | FIGURE 5.18 — Four Factor Interaction Response Table 65

35 | FIGURE 5.19 - Comparison of main effects and most significant 66

interactions

36 | FIGURE 5.20 - AD interaction graph and response table 66

40 | FIGURE 5.24 - Transaction Processor error log, Taguchi design 69

41 | FIGURE 5.25 - The Internationalization Wizard user interface 72

43 | FIGURE 5.27 - Classes of the Internationalization Wizard 74

44 | FIGURE 5.28 - Proportion of methods covered per experimental run 74

45 | FIGURE 5.29 - Proportion of methods covered discounting initialization | 76 |

Functions

46 | FIGURE 5.30 - Comparison of DOE and Facade pattern 78

coverage results on per-run basis

47 | FIGURE 5.31- Comparison of DOE coverage results based 78

on the estimated average response

49 | FIGURE 5.33 - Classes of the Contact Mediator 81

50 | FIGURE 5.34 - Comparison of coverage results including the Mediator 81

pattern

51 | FIGURE 6.1 - The Invoice Application interface 83

53 | FIGURE 6.3 — Number of Methods per Pattern Type 90

Vil

56 | FIGURE 6.6 — Pattern overlay diagram for the Invoice Application 92

57 | FIGURE 6.7 - Pattern Coupling through shared class 2 93

58 | FIGURE 6.7 — Revised Pattern Block Diagram showing the expected 101

pattern node, pattern coupling, and class node method coverage

60 | FIGURE 6.9 — Parallel/Series Pattern Block Diagram 105

Vili

I Introduction The process of software testing is challenging and may occur at many stages of design and development, from unit testing to integration testing to acceptance testing Following delivery, additional modifications may call for regression testing At each of these levels we may apply a variety of black-box and white-box tests Regardless of the

method chosen, we cannot test every possible combination of inputs Instead, we are

forced to select a subset of possible test cases

The generation of test cases is a problem in itself Hopefully the set of test

cases we select maximizes the number of errors we discover Several coverage

criteria have been proposed to insure that we exercise all statements, all branches, all conditions, or some combination of these Adding to the complexity, programs

developed under an object-oriented paradigm have characteristics not present in

procedural software The fundamental unit of an object-oriented program is the class When we test a class we are actually testing an instantiation of that class which is an object In testing object-oriented software, we must be aware that the state of an

object at any time depends on the sequence of messages

received by the object up to that point in time As these messages are passed by

method calls, the number of methods calls and the order in which they occur are an additional consideration

Two coverage metrics useful in the analysis of method calls among classes are control flow and data flow Control flow is primarily concerned with the branch and loop

structures of a program while data flow focuses on the bindings between variables and their values and how the variables are to be used These are often referred to as

definition/use or DU pairs Much research has been done pertaining to path selection criteria which are based on control flow or data flow techniques Of particular interest to researchers are strategies by which the generation of test cases may be automated

Several such methods are described in this paper and most involve complex algorithms

The use of patterns in software development has become more attractive given the rise in popularity of development tools utilizing a model driven architecture (MDA) philosophy If a given piece of software is designed from a collection of standard design patterns, we should be able to test that software based on the designer’s intent as captured

by the design patterns With any popular methodology there can be a tendency to over- promote the inclusion of such devices but experience has shown that, when judiciously applied, design patterns can have a positive effect on both the development and quality of software

It would be beneficial if similar positive influences were realized in the testing of pattern based software Here we examine the utility of design patterns in the practice of software testing In our investigation we employ a simpler metric of method coverage to gauge the

efficiency of the testing process Method coverage for two test procedures, statistical experimental design and the reliability block diagram, is examined Under certain

conditions, the presence of design patterns is found to provide a comparable degree of method coverage with significantly less effort We are further able to exploit these

patterns by constructing a new type of system analysis diagram based on design patterns and method coverage

II Object-Orientation From a conceptual viewpoint, object-orientation is a view of our environment in terms of separate objects An object is a concept, abstraction, or thing with distinct boundaries and meaning in an environment Under this paradigm, the various

considerations that pertain to our environment may be put into distinct categories The construction of these categories is facilitated by the fact that every object has attributes that we are knowledgeable of through use As a mundane example, consider the area of transportation We may intuitively construct broad categories of land, water, and air transportation based on attributes that we know conveyances in these mediums typically possess For instance, we may assume that a land vehicle has an attribute wheels while a water vehicle would not and that a vehicle for air travel has an attribute of wings In object-oriented terminology these categories of vehicles are known as classes and serve

as templates for the objects belonging to that class An object’s type is determined by the template (class) used to construct (instantiate) the object

From a programming point of view, object-orientation involves program objects These are conceptual objects modeled in the program code and represent entities in the real world which can be uniquely identified An entity modeled by an object is usually a

person, a place, a thing, or a temporal event As previously described, an object will be of

some class type The class describes a group of objects having common attributes,

behavior, and semantics Each program object in the group has a unique identity, state and behavior Identity is an intrinsic property which allows different instantiations of the

same class to be distinguishable, even if the resulting objects have the same state The state of an object is captured in the contents of its data fields These values may be of a primitive type or may represent a reference to another object The behavior of an object is defined by a set of methods which provide the functionality of the object and allow for the initialization, inspection and modification of the object’s state

In addition to serving as a mold for objects, classes enforce encapsulation and information hiding A typical user of a class (a client) is not concerned with the inner workings of the class but instead, requires only information on how to use the class Typically, communication with a class is accomplished through the class interface which

is a subset of the methods belonging to that class Specifically, those methods which a user may invoke Similarly, a user’s access to the data fields of a class may be restricted

to some subset of the total collection of fields Each method or field can be assigned a specific level of accessibility with an access modifier such as public, private, or

protected Those of private status are only available from within the current class

Protected status widens availability to classes in the same package (a group of classes) and to subclasses Greatest visibility is accorded by the public modifier which opens availability to all classes in all packages

The object-oriented philosophy allows us to derive new classes from existing

classes Such a derivation is called inheritance In object-oriented parlance, a subclass is

a class which inherits the properties of another class called the superclass The subclass

not only inherits accessible data fields and methods from its superclass, but it may also

add new data fields and methods This concept may be illustrated by a simple example Consider a banking scenario in which checking accounts and savings accounts are both

instances of a bank account Each of these accounts has distinct characteristics but they

also have common attributes Each can inherit the shared attributes from a more general account type, reusing the functionality of the superclass As demonstrated in Figure 1.1,

if a checking account and a savings account belonged to the same owner, a checking

account object and a savings account object could share common customer bank account information such as the customer’s name, address, and identification number There is no

need to duplicate this information in each subclass Instead, each could inherit that

common information from the general bank account class Similarly, a premium savings account and a passbook savings account are both more specific types of a general savings account Note that as you move up the inheritance hierarchy the objects are generalized and as you move down the hierarchy they specialize

FIGURE 1.1 — An Inheritance Hierarchy

As we have seen, the mechanism of inheritance makes it possible to reuse object- oriented code and add capabilities without having to change the existing code A subclass

is a specialization of its superclass; every instance of a subclass is an instance of its superclass, but not vice versa As a result, an object of a subclass can be used by any code designed to work with an object of its superclass This is known as polymorphism, from

the Greek word meaning “many forms” Polymorphism typically applies to objects that

are part of the same inheritance hierarchy; all of the objects within a hierarchy can potentially respond to requests that objects earlier in the hierarchy could respond to In Figure 1.2, we see some representative code for the rightmost branch of the banking hierarchy of Figure 1

ublic class bankTest{

public static void main(String[] args){

m(new bank_ account());

[class savings_account extends bank_account{

public String toString{)<

return “savings account";

}

lass passbook_account extends savings_account{

FIGURE 1.2 -— A Branch of the Banking Hierarchy

When the method m(Object x) is invoked, the toString method of argument x is called Now, x may be an instance of the bank_account, savings account,

passbook_account, or Object classes Each of these will have its own implementation of

the toString method which returns a string representing that object This implementation may be explicitly furnished by the class, as in savings account If none is provided, the

internal hexadecimal memory address (hashcode) of the object is returned Which

implementation is used will be determined dynamically at runtime by the Java Virtual Machine (JVM) This is known as dynamic binding, which works as follows: Suppose an

for

object o is an instance of classes C,,C,, ,C,_;,C, where C; is a subset of é, +

i=l to n-1, as shown in Figure 1.3

FIGURE 1.3 — Dynamic binding hierarchy

In the Figure, c,, is the most general class and c, is the most specific In Java, c,, would

be the Object class which is the root of all class hierarchies If object O invokes a method m, the JVM searches for that method in classes đ¡,€¿, , C„_¡; €ạ , in that order, until it is found Once an implementation is found the search terminates and the first

implementation found is invoked When the main method of the bankTest class executes, the result is to instantiate an object of bank_account, savings account, and

passbook_account, and to call their toString methods The output of this given in Figure 1.4

bank_account@119c082 savings account

savings account

FIGURE 1.4 - Outnut of bankTest

Since passbook_account is a subclass of savings account, an instance of

passbook_account is also an instance of savings account Hence, passbook_account inherits its toString method from savings account Bank_account, which provides no implementation of its own will actually inherit the toString method of the Object class In the Java programming language the Object class is, implicitly, the root of all hierarchies

Finally, no discussion of object-orientation would be complete without addressing

the concepts of association, composition and aggregation Association refers to a

relationship or interaction between two classes As we have seen, one possible

association is through the mechanism of inheritance Remembering that a class may be the origin of many objects, multiple instances of a class may participate in an association The number of instances participating in the relationship is the cardinality A common illustration uses the classes Student, School, and Course as there are two easily

understood relationships between these three entities In general

e Many Students attend a School

e A Student enrolls in many Courses

These associations are illustrated, using UML notation, in Figure 1.5 [4] The verbs attends and enrolls describe the relation between the classes The words Many and A

refer to cardinalities which are indicated at the ends of the links in the diagram Here, the

integer 1 corresponds to A and the symbol * corresponds to Many Specific ranges may

also be indicated For example, on the link between Student and Course, 1 * indicates a

minimum cardinality of | and a maximum cardinality of Many, that is, any number of

students 1 6 indicates a minimum cardinality if 1 and a maximum cardinality of 6, that

is, a Student must enroll in at least 1 Course but may enroll in no more than 6 Courses

FIGURE 1.5 — Class Associations and Cardinalities

Both aggregation and composition are special kinds of associations Aggregation

is used to represent ownership or a whole/part relationship, while composition represents

a stronger form of ownership An aggregation of objects is created when one object (the whole) contains in its encapsulated data one or more other objects (the parts) With composition, we have coincident lifetime of a part with the whole The composite object has sole responsibility for the disposition of its parts in terms of creation and destruction

In implementation terms, the composite is responsible for memory allocation and de- allocation Furthermore, the cardinality of the aggregate end may not exceed one; i.e., it

is unshared An object may be part of only one composite at a time If the composite is destroyed, it must either destroy all its parts or else give responsibility for them to some other object A composite object can be designed with the knowledge that no other object will destroy its parts Figure 1.6 represents an automobile object as the aggregation of

interior, body, and power-train objects As illustrated, a component of an aggregation may itself be an aggregation

FIGURE 1.6 — Aggregation of Objects

While object-orientation enhances our power to model our environment and increases the option of code reuse through its ability to combine existing classes, the interaction of these classes further complicates the already difficult task of testing In recent years, software engineers have adopted the philosophy of design patterns in an attempt to tame this explosion of classes

10

III Design Patterns

As we observe the world around us, we make inferences about its structure by

abstracting cause and effect As we have seen, this process of abstraction lies at the root

of the object-orientation paradigm However, if we also employ these observations in documenting solutions to recurring problems obtained under different conditions, we may construct empirical rules representing regularities of behavior Such rules are often referred to as “patterns” These patterns are not invented, but discovered through a

process of inquiry and observation It is noteworthy that humans have the ability to observe such patterns within environments that can vary from social to technological By accumulating these patterns over time, a discipline working within these environments can construct a common body of knowledge If properties for combining these patterns are established, a pattern language is formed which can act as a framework for design activities within that discipline

The founding father of the study of patterns in modern design is Christopher

Alexander [1] whose work in this area was directed towards the study of patterns in architecture and human communities His observations were motivated in part by a desire

to determine:

e What is present in a high quality design that is not present in a low quality

design?

11

e What is present in a low quality design that is not present in a high quality design?

e How do you duplicate a high quality design repeatedly?

He attempted to answer these questions by searching for commonalities in the designs of buildings, towns, cities, and the spaces we live in

To Alexander, a pattern accomplishes two things: it describes a problem that occurs over and over again in our environment; and it provides the core of a solution to that problem in such a way that one can use that solution a million times over, without ever doing it the same way twice His pattern philosophy was set forth in a set of two books, A Pattern Language [1] and The Timeless Way of Building [2], with the former volume containing a catalog of 253 architectural patterns In A Pattern Language, the presentation of the patterns was from largest to smallest, with patterns for regions and

towns given first, then neighborhoods, then buildings, rooms, alcoves, and finally the

details of the construction itself Alexander’s intent was to imply a definite ordering and inherent structure to his patterns His belief was that no pattern was an isolated entity and that each could exist only to the extent that it was supported by other patterns A given pattern is upheld by the larger patterns in which it is embedded and, at the same time, by the patterns that surround it at the same level Consequently, a proper pattern language will provide a framework for relating the patterns along both horizontal and vertical planes and the combinations may occur in an infinite number of varieties

Over the last decade design patterns have become the lingua franca of object- oriented software development Patterns are used by software developers and system architects to describe a general solution to recurring design problems Subsequent

12

developers can then apply the pattern based solution to their specific problem, hence design patterns provide an avenue towards effective code reuse In adapting Alexander’s work, Coplien [9] recommends a philosophy of commonality and variability analysis as a tool for software design In his thesis he recognizes that abstraction techniques all share some common principles Commonality seeks the structure that is unlikely to change significantly over time while variability captures those aspects of the design that are likely to be altered or transformed The intent is to contain the variations in independent classes, thereby allowing for future transformations with minimal effects on the code One way software designers try to achieve this is to contain variation within abstract

classes and then see how these classes relate to each other Over time, certain patterns in

the relationships have been observed to occur repeatedly If a given software product is considered to be of superior quality it is only natural that designers would want to capture certain characteristics of this software in succeeding projects Gamma, Helms, Johnson, and Vlissides have attempted to do just that in their seminal work on design patterns [16] These authors, known as the Gang of Four in the design pattern community, cataloged twenty three software design patterns and categorized them as creational, structural, or behavioral in nature

Creational design patterns are useful in abstracting the instantiation process In other words, they help make a system independent of how its objects are created and composed These patterns encapsulate knowledge about which concrete classes the

system is using by concealing how instances of these classes are created and put together

All the system knows about the objects is their interfaces as they are defined by abstract classes Accordingly, the creational patterns provide flexibility in what objects are

13

created, who creates them, how they are created, and when they are instantiated

Creational patterns allow one to make a design more flexible but not necessarily smaller

Structural patterns are concerned with how classes and objects are composed to form larger structures On a class level, structural class patterns use inheritance to

compose interfaces or implementations For example, consider how a designer in Java may draw from the characteristics of two or more classes through a form of multiple

inheritance While multiple inheritance, per se, is not allowed in Java some of its effects

may be achieved by allowing an object to inherit method signatures from multiple super- classes, with the caveat that the inheriting object is responsible for the implementation of those inherited methods This is made possible by encapsulating the method signatures in

an interface construct From an object point of view, structural object patterns represent methods of composing objects to achieve new functionality The increased flexibility of

object composition comes from its ability to alter a composition at run-time, a procedure

not possible with static class composition

Behavioral patterns are concerned with a system’s algorithmic behavior and the assignment of responsibilities between objects in the system, that is, the patterns of communication between them Again, we can examine these patterns from the point of view of either a class or an object Behavioral class patterns distribute behavior between classes through inheritance, whereas, behavioral object patterns use object composition Some types of behavioral object patterns are concerned with encapsulating behavior in an object and assigning requests to it

Regardless of the type of pattern, Gamma, et al, recognize that each has four

essential elements: the pattern name, the problem, the solution, and the consequences

14

J, The pattern name acts as a handle to describe a given design problem, its solutions, and consequences in the space of a few words, i.e., a brief description

2 The problem describes when to apply a particular pattern by describing the context of the design problem This can range from the current object

or class structure to a list of conditions which must be met before a certain pattern can be applied

3 The solution provides a template for a design pattern which includes an abstract description of the design problem and its general arrangement, a framework so to speak, of the objects and classes which are used to solve the problem This includes their roles in the pattern, how they collaborate, and the distribution of their responsibilities

4, The consequences are the results of applying the pattern and the trade- offs in memory requirements or run-time that may occur as a result

In short, a software design pattern is a description of communicating classes and objects that are customized to solve a general design problem in a specific context

Even a collection of the size and scope of that presented by the Gang of Four is not all inclusive as patterns specific to a given problem domain will exist as well as those

considered as “standard.” In any event, recognizing the existence of such patterns a priori and incorporating them into our designs enhances the reusability of our software

However, what actually constitutes a pattern, and even how one decides to classify a

pattern, depends upon one’s point of view, the level of abstraction the designer chooses to work at, and the language chosen by the software designer Metsker [31], for example,

15

considers the adapter, fagade, and bridge patterns to be forms of interface patterns while Gamma, et al, refer to them as forms of a structural pattern This is partly because

Metsker works in Java, which contains an interface class construction, while Gamma’s

patterns are constructed in C++ or Smalltalk, neither of which has an interface class per

se The choice of language may also affect what can and cannot be easily implemented For the purposes of this paper and the examples we discuss, Java will be the language of

choice

If this seems to you to be a “cookie-cutter” approach to solving a design problem, you are not alone There is a vocal segment of the software design community which takes the view that software design patterns, as they are currently understood, are nothing more than software templates that programmers can cut and paste into their projects In truth, this is probably how they are most often applied in practice and Alexander himself was not unaware of this possibility Regarding his collection of architectural patterns he said, “The fact that it is published as a book means that many thousands of people can use

it Is it not true that there is a danger that people might come to rely on this one printed language, instead of developing their own languages, in their own minds?” [1]

Nevertheless, to think of them as mere templates does not do justice to Alexander’s creativity At its core, his intent was for design patterns to supply a framework to support

a solution to a design problem and a body of common terminology which would allow that framework to be transferred to multiple instances of the same problem The

individual designer bore the responsibility of covering the framework with the details needed to tailor that solution to his particular occurrence of the problem

16

For example, let us suppose that a homeowner contracts with a builder to adda

garage to his house and that he desires a door for passage between the house and the garage The framework implied is that the door will be solely for pedestrian traffic and will be for utilitarian use only The body of common knowledge implied is that there are certain widths and heights expected and that the door will not have to be as aesthetically pleasing as a main entrance The builder may then tailor the solution in a multitude of ways with details such as color, type of material, type of lock, presence or lack of

windows, direction in which the door will open, etc

Now, while it is true that the builder may go to a construction supply house and purchase a door pre-hung in a frame, these “template-doors” in no way negate the power and usefulness of the initial abstraction process The body of common knowledge that passes between the homeowner and the builder during the design phase is made possible

by a mutual understanding of what is required for this particular type of door in this particular location, that is, a pattern exists for a pedestrian-utilitarian door To tie this example in with Gamma’s format, our four essential elements are:

1 The pattern name (brief description): pedestrian-utilitarian door Provides access for pedestrian traffic only Aesthetic appearance is not a consideration as door will be for utilitarian use only

2 The problem (context): A door is required for passage between the interior of a house and an attached garage

3 The solution (template for a design): A wooden door of 36 inch width and of standard 7 foot height will be used The door will open inward to the left, will have a window, and a dead-bolt lock for security

17

4 The consequences (tradeoffs): Wood material will require more frequent painting

than steel or fiberglass The presence of a window may reduce security somewhat

18

IV Software Testing

The process of testing includes elements of both verifiction and validation It is often thought of as checking to see that all the pieces of a system fit together correctly and that the system produced in the end is the one you set out to build It is sometimes said that you are trying to determine if you have built the right thing (validation) and that you have built the thing right (verification) Testing is not a single phase of the software lifecycle but occurs at every stage of system design and construction There are several generally accepted levels of software testing Unit testing exercises a unit in isolation from the rest of the system A unit is typically a function or small collection of functions and is small enough to test thoroughly, if not exhaustively Unit tests are normally white box tests since the small size of units allows a high level of code coverage It is generally

easier to locate and remove bugs at this level of testing

When software units are brought together, the integrated system frequently fails, regardless of our efforts at unit testing Jntegration testing exercises several units that

have been combined to form a module, subsystem, or system Integration testing focuses

on the interfaces between units, to make sure the units work together The nature of this

phase is also white-box, as we must have a certain knowledge of the units in order to

recognize if we have been successful in uniting them in a module In the opinion of some researchers, developers often apply one of three methods at this stage: top-down, bottom-

up and 'big bang’ Top-down combines and tests top-level routines that become the test

19

‘harness’ or 'scaffolding’ for lower-level units Bottom-up combines and tests low-level units into progressively larger modules and subsystems 'Big bang’ testing, seemingly the most prevalent integration test method, is waiting for all the module units to be complete and then trying them all out together Integration tests can rely heavily on stubs or drivers A stub substitutes for a finished subroutine or sub-system and might consist of a

function header with no body, or may read and return test data from a file, or obtain data

from a driver A desire to avoid stub creation and the development of drivers often leads

to the use of ‘big bang’ testing

External function testing is a black box test which exercises the functionality of

the system In other words, does the software perform according to specifications? This is sometimes known as an alpha test A stronger version of this phase may attempt to replicate the actual user environment, including hardware, database setup, network

infrastructure, etc If so, some refer to these functional tests as system testing

Acceptance testing or beta testing is an exercise of a completed system by a group

of end users Despite the sense of closure implied by the word “acceptance”, the tester’s work is not done The system will almost certainly receive modifications at some point as additional functionality is requested or as developers debug the software Regression testing is performed following modifications to confirm that the changes are correct and

do not adversely affect other system components

Throughout these proceedings, the tests we perform are traditionally considered to

belong to one of two broad categories, black-box tests or white-box tests Black-box tests

depend on the interface to a unit without regard to its internal structure In this approach our test data are derived from the software specifications and our test cases are valid

20

combinations of input conditions Exhaustive input testing would require the use of every

possible combination of inputs This is obviously impractical As a result, we are unable

to test a program to the point that we can guarantee that it is error free We can only show, to a degree of statistical confidence, that it will accomplish the purpose for which it was intended

The focus of white-box testing is on the internal structure of the software Test data are derived from an examination of the program’s logic Control flow testing and data flow testing are two broad categories of this technique In either strategy, our focus

is on covering some set of paths within the flow of program control or data An analogy

to the problem of exhaustive input testing in the black-box approach is the problem of exhaustive path testing, i.e., executing, via test cases, every possible path through a program Again, this is impractical since the number of distinct logic paths between two points in a program can be extremely large and can be further exacerbated by the loops and conditional tests which may be present In the real world, every logic decision is not independent of every other decision This raises the potential of pruning the number of test cases needed by considering categories of test cases as opposed to every individual case Yet, even if total path coverage could be achieved, we could not guarantee that the

program meets the desired specification If our goal had been to produce a descending-

order sorting routine but we instead produced an error free ascending-order sorting routine, exhaustive path testing would not be very helpful Moreover, exhaustive path testing is not useful in the detection of missing paths

Because exhaustive testing is impossible, test case design is important We are forced to select a subset of test cases so we should attempt to select a subset which

21

maximizes the number of errors we discover In general, randomly selecting

combinations of inputs is the least effective strategy A reasonable strategy is to use some combination of black-box and white-box methods Each technique has its own strengths and weaknesses and either one may find an error the other misses Myers [35]

recommends developing test cases using black-box methods and then supplementing these with white-box methods as necessary Some combination of the following

techniques may be useful:

Condition coverage Decision-condition coverage Multiple-condition coverage

White-box testing is concerned with the degree to which test cases cover the logic

of the program A perfect white-box test would exercise every possible path in the program but this is obviously not realistic in a program containing loops since such a test would also have to execute every possible combinations of iterations of each loop Conceding that the traversal of all paths is not practical, we could instead choose our test cases to insure that we at least exercise every statement in the program However, this strategy too has its weaknesses Consider the short code segment in Figure 3.1, described

by Myers [35] We could execute every statement by applying the test case { A= 2, B=

0, X = 4 } at point a in the code This results in following path a,c,e

22

ublic void Foo{int A, int B, int X) {

FIGURE 3.1 — Flowchart for Path Traversal in White Box Testing

There are a couple of potential problems worth noting here Forcing a statement

to execute will not necessarily detect errors in the logic of that statement For example, the same path would result if the statement “A>1I and B=0” had instead been coded as

“A>I or B=0” In addition, using this test case, we completely miss a path (a,b,d) over

which the value of variable X remains unchanged This may or may not be meaningful

We could cover this missed path by employing a decision coverage or branch coverage

strategy

A decision coverage or branch coverage criterion requires that each decision statement assumes both a True and a False state at least once during the execution of the

test, i.e., we want each branch to be traversed at least once This technique will also often

satisfy statement coverage since every statement is on some branch path or some path entering the program Possible exceptions would be programs with no decisions or programs or methods having multiple entry points which may allow us to bypass some

statements To cover these exceptions, we should plan our tests to ensure that each

23

decision have a true and a false outcome and that every point of entry into the program or method is invoked at least once Multiple decision statements such as select(case) in Java require care when planning a test case Following this rule of thumb, we could achieve both decision coverage and statement coverage

A stronger test criterion is condition coverage To satisfy this criterion, we would need enough test cases to ensure that each condition in a decision statement takes on all possible outcomes at least once As with decision coverage, this technique will not guarantee that all statements are executed so you should plan accordingly if this is

desired Table 3.1 is a summary of the conditions present and the combinations of

outcomes we must consider in Figure 3.1

TABLE 3.1 — Conditions for Path Traversal in White Box Testing

Submitting test cases {A = 2, B=0, X =4} and {A=1, B=1, X= 1} at point a will provide condition coverage for this example The first test case will follow path a,c,e while the second case follows path a,b,d Condition coverage may or may not provide decision coverage For example, If we had instead submitted the test cases {A = 1, B = 0,

X = 3} and {A =2, B= 1, X = 1}, we would still cover the above outcome combinations but both of these cases would follow only path a,b,e Hence, we would miss a True result

in the decision at point a and a False result in the decision at point 5 To rectify this shortcoming, we may opt for decision/condition coverage

24

Decision/condition coverage requires a sufficient number of test cases such that

each condition in a decision takes on all possible outcomes at least once, each decision takes on all possible outcomes at least once, and each point of entry is invoked at least once As strong as this sounds, as with any criteria, there are pitfalls to consider Consider the flowchart in Figure 3.2 in which the compound decisions of Figure 3.1 are separated into individual decisions This more detailed analysis of the code serves to illustrate how the evaluation of one condition may mask the evaluation of another For example, if

“A>1” in the and condition were False, the condition “B = 0” may not be exercised As a result, errors in conditional logic may not be exposed by the condition or the

condition/decision criteria

With the awareness that some condition outcomes may mask other condition outcomes by preventing their evaluation, we could attempt to force our test cases to execute all possible condition outcomes in every decision and, as before, ensure that each point of entry is invoked at least once This is sometimes referred to as a multiple-

condition criteria Consider the pseudo-code of Figure 3.3 for printing the position for each occurrence of the name “Smith” in a given list of names

25

The four situations which logically would need to be tested under a multiple-condition criteria are given in Table 3.2 To exercise the decision 1, decision 2 outcomes of TF and

TT, we could submit a nonempty list with at least one name of “Smith” and one name

other than “Smith” For the outcomes FF and FT, when the condition P > L is satisfied

all outcomes of decision 2 will be masked These last two outcome combinations may be collapsed to a single condition, i.e., can we get out of the loop The point is that, as a practical matter, we won’t be able to exercise all possible logical combinations

TABLE 3.2 — Multinle Condition Criteria

To summarize at this point, note that a set of test cases which satisfies the

multiple-condition criteria will also satisfy the decision coverage, condition coverage, and decision/condition criteria Table 3.3 contains a brief comparison of these path coverage criteria

Not only is it impractical to attempt to reproduce every possible input and output combination, as we have just seen, it is sometimes even impossible to reproduce some combinations In a way, this can work to our advantage as it reduces the number of test

cases necessary If a particular path is implied by a truth Table but in fact cannot be

executed by the program why spend the effort to create a test case for it? Reducing the number of test cases is the intent behind the method of equivalence partitioning, a black box test strategy

27

Decision coverage Force each decision to have a True and a

Faise outcome at least once

Condition coverage Force each condition in a decision to take

on all possible values at least once

Invoke each point of entry at least once

Decision/condition coverage Force each decision to have a True and a

Faise outcome at least once

Force each condition in a decision to take

on all possible values at least once

Be aware that certain condition outcomes may prevent other condition outcomes

Invoke each point of entry at least once

Multiple condition coverage

Write sufficient test cases to insure, as

much as practical, that all possible combinations of condition outcomes in every decision are exercised

Invoke each point of entry at least once

Our desire is that each individual test case invoke as many different input considerations

TABLE 3.3 — Comparison of Path Coverage Criteria

as possible This allows us to partition the input domain into a finite number of

equivalence classes such that an individual test case within a class is representative of all test cases within that class To apply this strategy we must first identify the equivalence

classes Then, we must define the test cases

Generally, we may identify two broad categories of input types from which we can form equivalence classes: valid inputs and invalid inputs Some commonly accepted guidelines are given in Table 3.4 Typical test cases for each equivalence class would

include values at the boundaries and values at the center of the class

28

If input specifies possible equivalence classes type of class

© a<x<b e valid

where x is the number of input values

each value is processed

differently

A specified “must be” or a ¢ Condition satisfied e valid

TABLE 3.4 — Guidelines for Equivalence Classes

Returning to our search for the name Smith, we may identify two equivalence

classes at each of the two decisions present in Figure 3.3 Decision | deals with an integer number of values, specifically, the number of positions in the list where the name Smith may be found A valid equivalence class would be to let the name’s POSITION range from n to nt+k Here n = 1 and k = listsize —1 Invalid input equivalence classes for

POSITION are not a consideration since the name Smith could not be submitted outside

of the boundaries of the list A possible test case for decision 1 would be to submit a list containing Smith in the first position, in a middle position, and in the last position

Decision 2 treats a Boolean condition which implies the equivalence classes name found and name not found Test cases would be to submit a list containing Smith and a list not containing Smith

A strategy sometimes referred to as sub-domain testing involves a partitioning of the input domain into mutually exclusive sub-domains This technique is helpful if the partitioning tends to isolate potential errors within individual sub-domains Every-

29

statement coverage and every-branch coverage are not sub-domain tests We do not have mutually exclusive sub-domains related to the execution of different statements or

branches On the other hand, every-path coverage is a form of sub-domain testing since the sub-domain of test cases that execute a particular path through a program is mutually exclusive of the sub-domain for every other path Often errors occur at the boundaries separating these sub-domains Decision statements can be a source of boundaries between input domains The search for potential errors at theses boundaries is sometimes called, appropriately enough, boundary testing Other terms for this technique are boundary- value analysis or domain testing

In decision 1 of Figure 3.3 the true sub-domain for values of POSITION has an open boundary at LISTSIZE which places the value of LISTSIZE in the false sub-

domain If the decision had been coded as POSITION < LISTSIZE, then the LISTSIZE

value would be incorrectly placed in the true sub-domain Another possibility for error is the shifting of a domain boundary

In the terminology of boundary testing, on tests are in the true input domain and off tests are in the invalid domain An on point is a point on a domain boundary or as close to the boundary as possible while satisfying the boundary conditions If the domain boundary is open, then an off point is an interior point of the domain within an ¢

neighborhood of the boundary If the domain boundary is closed, then an off point is an exterior point just outside the boundary, not satisfying the boundary conditions We

would like to ensure that the actual boundary between two sub-domains is as close as

possible to the specified boundary Intuitively, we should select input values at (on) the boundary and on either side (off) of the boundary The usual practice for the off tests are

30

to select values both near the boundary and far from the boundary, i.e., well into the sub-

domain Beizer [5] is more economical, recommending just two tests per boundary inequality using one on point and one off point input value He observes that all the errors that can occur at a one-dimensional boundary between domain A and domain B are: 1) Closure Error The boundary is open when it should be closed or vice versa The on point test will catch this error because the input value will receive domain B processing when it should have received domain A processing, assuming no coincidental

correctness at the boundary

2) Boundary shifted left The off point should have received B domain processing but instead received A domain processing

3) Boundary shifted right The on point should have received A domain processing but instead received B domain processing This test cannot distinguish between a closure error or a right-shift error but we do know there is an error

4) Missing boundary Both the on point and the off point values receive the same

processing but we know the processing should have been different

5) Extra boundary An extraneous boundary divides the original domain into two sub- domains The presence of this additional boundary must result in a distinguishable difference in processing otherwise we will not be able to detect it

31