BVA is a widely used black box testing technique that consists of testing the SUT with values on and off an equivalence class boundary under the assumption that these are bug-prone regions. The aim is to detect domain faults, that is, applying a correct function to the wrong data. In this analysis, we will circumscribe to the most common use of BVA, which assumes that each boundary segment or border is defined by a linear predicate on a single variable forming a convex set (see Figure 5.19). These properties are the ones that allow for each test parameter to be varied independently from the oth- ers. Each border is either closed or open. A domain boundary is closed with respect to a domain if the points on the boundary belong to the domain and open if the points in it belong to another domain. A closed border is formed by a value and a relational expression with a ≤, ≥, or = operator. An open border results from a value and a relational expression with a <, >, or ≠ operator. These restrictions are not restrictions in the sense that you cannot use the techniques described if they are violated, but in the sense that if you apply them to such cases, tests are unlikely to be productive [13].
Other forms of BVA include domain testing [46,217], which looks at the execution paths to derive the partition of the SUT input domain and allows linear relationships among the domain variables.
The basic question we are trying to answer here is whether we should use BVA values in interaction testing or first establish confidence that the partition is indeed an equivalence class and there are no boundary prob- lems and then test for interactions using a nominal value from it.
While a definite answer to this question has proven elusive without mak- ing assumptions to the point of making it too restrictive for any practical use, we can analyze the issues and trade-offs for specific situations from two perspectives: the number of test cases required and what we get from them. For this, we will use the example in Figure 5.20, which describes a tree for the input domain defined by EC1= {(x, y)|2 ≤ x ≤ 10 and 1 ≤ y ≤ 7}
and EC2= {(x, y)|2 ≤ x ≤ 10 and 7 ≤ y ≤ 12} to illustrate the process. Figure 5.21 shows the input domain and the corresponding test cases for the four strategies outlined below and Figure 5.22 shows the number of test cases required as a function of the number of test parameters.
The analyzed testing strategies are
• BVA once each. In this strategy, we just make sure that each BVA value is included at least once in the test suite.
• BVA once each + All pairs EC. This strategy includes the above- described BVA once each followed by all possible pair of values at the equivalence class level.
Test points (c)
y
(a) 0 ≤ x ≤ k
0 ≤ y ≤ k 0 ≤ x + y ≤ k (b)
x
FIGURE 5.19 Nicest (a), nice (b), and ugly domains (c). The same number of test cases does not provide the same level of confidence in all cases. (a) Both variables are independent. (b) Functional linear relationship between x and y. Convex domain. (c) Nonlinear relationship between x and y. Concave domain.
108 ◾ Introduction to Combinatorial Testing
TestStrategy x
valuesBVA BVA
values EC1
EC2 1 y
2 6
10
11 0
(Outside lower limit)
(Outside upper13 limit) (Lower limit1
interior point)
(Lower limit8 interior point) (Nominal4
interior point)
(Nominal10 interior point) (Upper limit7
interior point)
(Upper limit12 interior point)
FIGURE 5.20 Should we test first for all values under each equivalence class and then use interaction testing at the EC1 and EC2 level or should we use combina- torial testing with all the values generated by BVA?
A = {(x,y) | 2 | | x | | 10, 1 | | y | | 7}; B = {(x,y) | 1 | | x | | 10, 7 < y | | 12}
14
y
x 1312
1110 98 76 54 32 10
0 1 2 3 4 5 6 7 8 9 10 11 12 13
Boundary “A”
BVA once each BVA + nominal All pairs All pairs EC Boundary “B”
FIGURE 5.21 (See color insert.) A two-dimensional convex region tested using different strategies.
• BVA + Nominal. In this case, we combine each of the BVA val- ues from each parameter with a nominal value from all the other parameters.
• All pairs. All possible pairs of BVA values are included in at least one test.
Testing all BVA values for two-way interactions in the example requires 40 test cases. Testing first for all equivalences using each value once fol- lowed by interaction testing will require 8 + 2 = 10 test cases.
Other things being equal, for 10 test parameters, the number of test cases required by the “BVA once each + All pairs EC” is a sixth of that required by the “All pairs BVA”: 33 versus 195. A more thorough strategy,
“Nominal + All BVA,” which tests all BVA values from each parameter with a single nominal value from each of the other test parameters, followed by
“All pairs of EC” values requires little more than half the values of “All pairs,” 113 versus 195, but four times more than the “BVA once each + All pairs EC strategy.” In this example, from the point of view of the economy of test cases, the best strategy would be to first establish the equivalence of each class and then test for interactions using a nominal value. But what are we giving up when we choose the most economical alternative?
3 equivalence classes per test parameter, all pairs
BVA once each
All pairs EC values
BVA once each + all pairs EC
Nominal + BVA
Nominal + BVA + all pairs EC
All-pairs BVA values 250
200
150
100
50
0 0 5 10
Number of test parameters
Test cases
15 20
FIGURE 5.22 (See color insert.) Number of test cases required by different test- ing strategies.
110 ◾ Introduction to Combinatorial Testing
Boundary errors refer to the manner in which the program processes values at boundaries, and some types of errors include:
• Closure: A boundary is closed or open when it should have been the opposite, such as when ≥ is used when > is the correct operator.
• Missing boundary: Distinction between values is not processed, such as when a conditional is missing that should be present to distin- guish between two classes of input.
• Parallel boundary shift: Boundary is uniformly shifted across values, such as x > 20, when x > 10 is the correct condition.
• Boundary tilt: Boundary is shifted because coefficients in a condition are wrong, such as y > 2*x, where the correct condition is y > x (picture how these would be seen on a graph).
The BVA once each strategy can consistently detect closure, missing boundaries, and border shifts for parallel boundaries but cannot be relied upon to detect boundary tilts or spurious border problems. Because of the combinations of two or more “off limits” values, input masking might occur at the extremes.
The BVA + Nominal strategy is also capable of consistently detecting closure, missing boundaries, and parallel shifts. The larger number of test points will contribute to expose boundary tilts and spurious boundaries but there is no certitude any or all will be found. The possibility of mask- ing is removed by using only one off-region value at a time.
Similar to the other two, the all-pairs strategy will consistently detect closure, missing boundaries, and parallel shifts. It will also detect spuri- ous relationships among variables of rectangular domains by testing all corners of the domain. The possibility of masking exists because there are test cases that include two out-of-limits values but this is compensated by the combination of the off-limit values with the nominal ones.
Based on the above analysis, the test designer could design, for exam- ple, a test strategy like the one shown in Figure 5.23 in which “off limits”
values from each parameter are combined with all EC values from the other parameter to test boundaries without risk of input masking due to the combination of two or more “off limits” values. The all-interior points establish equivalence of the proposed classes and combinatorial testing is applied at the EC level. Notice that in the present example, because of the
number of variables and values involved, the all-pairs test points coincide with some of the all-interior points.