The regression testing strategy of comparing new results with old ones char- acter by character is well suited for output consisting of text and integers.
When floating-point numbers are involved, the comparison is much more challenging as round-off errors are introduced, either because of a change of hardware or a permutation of numerical expressions in the program. We want to distinguish round-off errors from real erroneous calculations. One possi- ble technique for overcoming the difficulties with comparing floating-point numbers in regression tests is outlined next.
1. The output to the logfile with extension .v is filtered in the sense that all floating-point numbers are replaced by approximations. In practice this means replacing a number like1.45298E-01 by an output with fewer decimals, e.g., 1.4530E-01. Numbers whose round-off errors are within the approximation should then be identical in the output. The user can supply a function taking a real number as argument and returning the appropriate approximation in the form of an output string. One example is
def defaultfilter(r):
if abs(r) < 1.0E-14: r = 0.0 s = ’%11.4e’ % r
return s
The first statement replaces very small numbers, which often arise from round-off errors, by an exact zero. Other numbers are written with four decimals. Another filer, exactfiler, makes the same round off to zero, but otherwise the precision ofr is kept (s=’%g’ % r).
Thedefaultfilterfunction has some unwanted side effects. For example, it replaces the text ’version 3.2’by’version 3.200E+00’. One remedy is to apply the approximation only to numbers in scientific notation or other real numbers written with more than (say) four decimals. This is taken care of in the implementation we refer to.
2. Since the introduced approximation may hide erroneous calculations, an additional output file with extension.vdis included, where all significant floating-point numbers are dumped without any approximations and in a special format:
## some text number of floats float1
float2 float3 ...
## some text number of floats float1
float2
B.4. Verification of Scripts 721 float3
...
## some text
and so on. A specific example is
## field 1 7
1.345 3.45 6.9 9 8.999999 1.065432E-01 0.04E-01
## field 2 4
1.6 3.1 2.0 1.1
The idea is to create a tool that compares each floating-point number with a reference value, writes out the digits that differ (for example by marking differing digits by a certain color in a text widget), and computes the numerical difference in case two numbers are different from a string- based comparison. The stream of computed numbers are plotted together with the stream of their reference values (if the difference is nonzero) in a scrollable graph, which then makes it easy to detect errors of significant size visually.
In other words, this strategy divides the verification into two steps: first a character-by-character comparison of running text and approximate repre- sentation of real numbers, and then a more detailed numerical comparison of certain real numbers. Serious errors will normally appear in the first test.
The implementation of the outlined ideas is performed in two classes:
TestRunNumerics for running tests with approximate output of real numbers, andFloatDiff for reporting results from the detailed numerical comparison.
Class TestRunNumerics is implemented as a subclass of TestRun and offers basically two new methods: approx for approximating the normal output produced by its base class TestRun, and floatdump for running a test and directing the output to a file with extension .vd in the special format out- lined above. This allows for detailed numerical comparison of a chunck of real numbers. Theapproxmethod takes a filter for performing the approximation as argument. At the end of a test script employing an instancetestof class TestRunNumerics we can hence make the call
test.approx(scitools.Regression.defaultfilter)
which imples that the defaultfilter function (shown previously) in the Regression module is used as filter for output of real numbers.
Comparison of an output filemytest.vdwith reference results inmytest.rd is performed by thefloatdiff.py script (insrc/tools):
floatdiff.py mytest.vd mytest.rd
The floatdiff.py script employs class FloatDiff in the Regression module to build a GUI where deviations in numerical results can be conveniently investigated. Figure B.2 shows such a GUI.
Fig. B.2.Example of the GUI launched from thefloatdiff.pyscript. By clicking on a field in the list to the left, the corresponding computed results are shown together with the reference results and the their difference in the text widget in the middle of the GUI. Deviations in digits are highlighted with a color. A visualization of the differences appears to the right.
Example. Let us use the features of the approx method described above to make the test scriptcircle.verify from page 713 independent of round- off errors. To this end, we need to write the script in Python, using the TestRunNumerics class:
#!/usr/bin/env python import os, sys
from py4cs.Regression import TestRunNumerics, defaultfilter test = TestRunNumerics(’circle2.v’)
test.run(’circle.py’, options=’1 0.21’)
# truncate numerical expressions in the output:
test.approx(defaultfilter)
This test script is found insrc/py/examples/circle/circle2.verify. Running regression.py verify circle2.verify
results in a filecircle2.v where all floating-point numbers are written with only four decimals:
B.4. Verification of Scripts 723
#### Test: ./circle2.verify running circle.py 1 0.21 -1.8 1.8 -1.8 1.8
1.0 1.3195e+00
-2.7802e-01 1.6476e+00 -9.1367e-01 4.9135e-01 4.8177e-02 -4.1189e-01 1.1622e+00 2.9512e-01 end
CPU time of circle.py: 0.1 seconds on basunus i686, Linux
In addition, we want to generate acircle2.vd with exact numerical results.
To this end, we add a few Python statements at the end ofcircle2.verify:
# generate circle2.vd file in correct format:
fd = open(’circle2.vd’, ’w’) fd.write(’## exact data\n’)
# grab the output from circle.py, throw away the
# first and last line, and merge the numbers into
# one column:
cmd = ’circle.py 1 0.21’
output = os.popen(cmd) res = output.readlines() output.close()
numbers = []
for line in res[1:-1]: # skip first and last line for r in line.split():
numbers.append(r)
# dump length of numbers and its contents:
fd.write(’%d\n’ % len(numbers)) for r in numbers: fd.write(r + ’\n’) fd.close()
The resultingcircle2.vd file reads
## exact data 10
1.0
1.31946891451 -0.278015372225 1.64760748997 -0.913674369652 0.491348066081 0.048177073882 -0.411890560708 1.16224152523 0.295116238827
We can run the regression test by
regression.py verify circle2.verify
Thefloatdiff.py script will not launch a GUI if circle2.vd is identical to circle2.rd. To demonstrate the GUI, we force some numerical differences by changing the digit at the end of each line incircle2.vdto 0:
subst.py ’\d$’ ’0’ circle2.vd
Running
floatdiff.py circle2.vd circle2.rd
results in a GUI where the differences between circle2.vd and circle2.rd are visualized.