This book introduces a wide range of topics in combinatorial methods for software testing, sufficient for handling many practical challenges in software assurance. Most testers, however, will not face all of the types of test problems covered in this book, at least not on every project. Many test problems require a core set of methods, possibly with one or two spe- cialized topics. As with many subjects, one of the best ways to approach combinatorial testing is to start small; try the basics to get a feel for how it works, then supplement these methods as needed. This book is designed
www.Ebook777.com
for such an approach. Readers who are anxious to learn by applying some of the methods introduced here can use the following steps:
1. Read Chapter 1, to learn why combinatorial methods are effective and what to expect.
2. Read Chapters 3 and 4, for step-by-step approaches to input testing and configuration testing (as introduced in Section 1.2).
3. Download and install the Java program ACTS or another covering array tool (see Appendix D).
4. Develop a covering array of tests using ACTS or other tool, then run the tests.
After reading this chapter to understand why combinatorial testing works, readers can also review the two case studies in Chapter 2. These two testing problems are practical examples that illustrate the basics in situations which include many features of web application testing prob- lems. Following the steps above is really just getting started, of course, and readers doing practical testing will encounter problems that can be solved using methods from the other chapters. For example, thorough input domain analysis (Chapters 5 and 6) to select test values is essential for strong testing and assurance. But trying the steps listed above on one of your own small testing problems will likely make the rest of the tech- niques introduced in the book easier and more interesting to apply.
1.6 CHAPTER SUMMARY
1. Empirical data suggest that software failures are caused by the inter- action of relatively few parameter values, and that the proportion of failures attributable to t-way interactions declines very rapidly with increase in t. That is, usually single parameter values or a pair of values will be the cause of a failure, but increasingly smaller propor- tions are caused by 3-, 4-, and higher-order interactions. This rela- tionship is called the Interaction Rule.
2. Because a small number of parameters are involved in failures, we can attain a high degree of assurance by testing all t-way interac- tions, for an appropriate interaction strength t (2−6 usually). The number of t-way tests that will be required is proportional to vt log n, for n parameters with v values each.
18 ◾ Introduction to Combinatorial Testing
3. A mathematical construct called a covering array can be used to pro- duce tests that cover all t-way combinations. There is no “best” cov- ering array construction algorithm, in the sense of always producing an optimal array.
4. As with all other types of testing, the oracle problem must be solved—
that is, for every test input, the expected output must be determined in order to check if the application is producing the correct result for each set of inputs. A variety of methods can be used to solve the oracle problem.
5. Combinatorial methods can be applied to configurations of the SUT or to input values, or both. Figure 1.5 contrasts the two approaches to combinatorial testing. With the first approach, we may run the same test set against all t-way combinations of configuration options, while for the second approach, we would construct a test suite that covers all t-way combinations of input transaction fields. Of course, these approaches could be combined with the combinatorial tests run against all the configuration combinations.
REVIEW
Q: What is the Interaction Rule and why is it significant?
A: The rule states that most failures are caused by one or two parameters interacting, with progressively fewer failures caused by 3, 4, or more parameter interactions. So if all failures are caused by the interaction of a small number of parameters, then testing combinations of these values up to an appropriate level is likely to catch nearly all errors.
System under test Inputs:
Product Amount Quantity Pmt method Shipping method
Configuration:
Browser OSDBMS Server ...
Use combinations of input values in generating tests
Use combinations of configuration values with existing test suite
FIGURE 1.5 Combinatorial testing may be used on input values or configurations.
Q: What are the two primary ways in which combinatorial testing is applied?
A: Combinatorial testing may be applied to configuration settings or input values, or both.
Q: What is a covering array?
A: A matrix that covers all t-way combinations of variable values, for some specified value of t.
Q: What does the term strength refer to in the context of t-way testing?
A: The value of t, the number of test parameters for which all allowed combinations are covered.
Q: Does use of a 4-way covering array guarantee detection of all 4-way faults? Why or why not?
A: No, because in most cases some input space partitioning is required for continuous variables. For example, a “distance” value may range from 0 to 25,000 miles. It will be necessary to select a small number of possible values that are significant in the software under test, such as 0, 99, 100, 999, 1000 1001, 99,999 miles. If this selection of values misses some significant point in the code, the test may not detect an error.
Q: The size of a covering array grows exponentially in proportion to what variables? Logarithmically in proportion to what variable? What is the significance of these facts?
A: The array size is proportional to vt, where v is the number of values per parameter and t is the interaction strength, that is, value of t for t-way testing. However, the number of tests increases only logarithmi- cally with the number of parameters. As a result, the number of values per parameter is a limiting factor in combinatorial testing, but a large number of parameters will not generally be a problem.
Q: What are three ways of handling the oracle problem?
A: Crash testing, embedded assertions, and model-based test generation.
Q: For what value of t must a t-way covering array be generated to catch the following fault?
if (A < 10 && B > 0) {bad code}
else {good code}
A: A 2-way array is needed, so that at least one test contains a combina- tion of values for A and B that will trigger the branch to the bad code, assuming that appropriate equivalence classes have been established.
20 ◾ Introduction to Combinatorial Testing
Q: For what value of t must a t-way covering array be generated to catch the following fault?
if ((A < 10 || B > 0) && C > 90) {bad code}
else {good code}
A: A 2-way array is needed, because either A && C or B && C will cause a branch into the bad code.
Q: For what value of t must a t-way covering array be generated to catch the following fault?
if (A < 10 && B > 0 && C > 90) {bad code}
else {good code}
A: A 3-way array is needed.
Q: Suppose you are asked to test a program with 10 input parameters, with 100 values each, using a 6-way covering array. Can you generate a covering array for this problem?
A: In theory yes, but not as a practical matter, because the size of a t-way covering array is proportional to vt for parameters with v values, so in this case the number of tests would be proportional to 1006= 1012. The way to approach this problem is to do input space partitioning so that roughly 10 or fewer representative values are used for each parameter.
This method is discussed in more detail in later chapters.
Q: If all variables have the same number of values, the number of t-way settings of n variables with v values each is vt × C(n, t). But in most cases, variables have different numbers of values. For the following five variables with values shown, how many 2-way settings (value pairs) are there: a: 0,1,2; b: 0,1; c: 0,1,2; d: 0,1; e: 0,1?
A: 6 + 9 + 6 + 6 + 6 + 4 + 4 + 6 + 6 + 4 = 57. This is vavb+ vavc+ vavd+ va ve+ vbvc+ vbvd+ vbve+ vcvd+ vcve+ vdve, where vi= number of values for variable i.
21 C h a p t e r 2
Combinatorial Testing Applied
In Chapter 1, we introduced the interaction rule and noted that if all failures can be triggered by t-way combinations or less, then testing all t-way combinations is in some sense equivalent to exhaustive testing. We have referred to this approach as pseudoexhaustive testing [103]: clearly, it is not exhaustive because we do not cover all possible inputs, and issues such as timing and concurrency may be missed. As always, testing is vital but it cannot guarantee the complete absence of errors [36]. But the ability to test all combinations of (discretized) values, up to the interaction strength actu- ally involved in failures, provides a powerful assurance method. Can this approach really detect as many faults as exhaustive testing across discretized values? Although it will take years of experience with combinatorial testing to give a full answer, this chapter summarizes two studies of practical prob- lems where combinatorial testing did indeed find faults as well as exhaustive testing with discretized inputs, using only a fraction of the tests.