The regression.py script referred to in the previous section is just a sim- ple call to functionality in a module scitools.Regression. For example, the commandregression.py verifyis basically a call to the constructor of class Verify in the Regression module. In the following we shall explain some of the most important inner details of class Verify. The complete source code is found in
src/tools/scitools/Regression.py
Knowledge of the present section is not required for users of theregression.py tool. Readers with minor interest in the inner details of the regression.py tool can safely move to Appendix B.4.3.
ClassVerify’s constructor performs a recursive search after files in a spec- ified directory tree, or it can handle just a single file. The recursive directory
5 One can also copycircle.vtocircle.rmanually, butregression.py update is more general as it can perform the update recursively in a directory tree if desired.
search can be performed with theos.path.walkfunction. However, that func- tion terminates the walk if an original file is removed by the verification script, something that frequently happens in practice since verification scripts often performs clean-up actions. We therefore copy the small os.path.walk func- tion from the Python distribution and make it as robust as required. The function is called walk and for its details we refer to theRegression.py file.
The constructor of classVerifythen takes the form def __init__(self,
root=’.’, # root directory or a single file task=’verify’, # ’verify’ or ’update’
diffsummary = ’verify_log’, # logfile basename diffprog = None # for file diff .v vs .r ):
<remove old log files>
<write HTML headers>
# the main action: run tests and diff new and old results if os.path.isdir(root):
# walk through a directory tree:
walk(root, self._search4verify, task) elif os.path.isfile(root):
# run just a single test:
file = root # root is just a file dirname = os.path.dirname(file)
if dirname == ’’: dirname = os.getcwd()
self._singlefile(dirname, task, os.path.basename(file)) else:
print ’Verify: root=’, root, ’does not exist’
sys.exit(1)
<write HTML footers>
Execution of a single regression test is performed in the following function, where we check that the extension is correct (.verify) and grab the associated basename:
def _singlefile(self, dirname, task, file):
"""Run a single regression test."""
# does the filename end with .verify?
if file.endswith(’.verify’):
basename = file[:-7]
if task == ’update’:
self._update(dirname, basename) elif task == ’verify’:
self._diff(dirname, basename, file)
The purpose ofself._diff is to run the regression test and find differences between the new results and the reference data, whereas self._update up- grades new results to reference status.
def _diff(self, dirname, basename, scriptfile):
"""Run script and find differences from reference results."""
# run scriptfile, but ensure that it is executable:
os.chmod(scriptfile, 0755)
B.4. Verification of Scripts 717 self.run(scriptfile)
# compare new output(.v) with reference results(.r) vfile = basename + ’.v’; rfile = basename + ’.r’
if os.path.isfile(vfile):
if not os.path.isfile(rfile):
# if no rfile exists, copy vfile to rfile:
os.rename(vfile, rfile) else:
# compute difference:
diffcmd = ’%s %s %s’ % (self.diffprog,rfile,vfile) res = os.popen(diffcmd).readlines()
ndifflines = len(res) # no of lines that differ
<write messages to the log files>
<quite some lengthy output...>
else:
print ’ran %s, but no .v file?’ % scriptfile sys.exit(1)
For complete details regarding the output to the logfiles we refer to the source code inRegression.py.
In the previous code segment we notice that the execution of the*.verify script is performed in a method self.run. The differences between the new results (*.v) and reference data (*.r) are computed by a program stored in self.diffprog. The name of the program is an optional argument to the constructor. If this argument isNone, the diff program is fetched from the en- vironment variableDIFFPROG. If this variable is not defined, thediff.pypro- gram that comes with Python (in$PYTHONSRC/Tools/scripts) is used. There are other alternative diff programs around: Unix diff and the Perl script diff.pl (requires theAlgorithm::Diff package). You should check out these two and the various output formats ofdiff.pybefore you make up your mind and define your favorite program inDIFFPROG.
The simplest form of the runfunction, used to run the script, reads def run(self, scriptfile):
failure, output = commands.getstatusoutput(scriptfile) if failure: print ’Could not run regression test’, scriptfile The system command running the script requires the current working direc- tory (.) to be in your path, which is undesired from a security point of view.
A better solution is to prefix the script with the current working directory, done as usual in a platform-independent way in Python:
scriptfile = os.path.join(os.curdir, scriptfile) Theos.curdirvariable holds the symbol for the current directory.
When visiting subdirectories in a directory tree, we make anos.chdirto the currently visited directory (see the self._search4verify method later).
This is important for theself._diff and other methods to execute properly.
In the case wherescriptfile executes a code in a compiled language like Fortran, C, or C++, we first need to compile and link the application before
runningscriptfile. This additional task can be incorporated in alternative versions ofrunin subclasses ofVerify. For example, regression tests in Diff- pack [15] are located in a subdirectory Verify of an application directory.
Therunfunction must hence first visit the parent directory and compile the Diffpack application before running the regression test. Here is an example on such a tailored compilation prior to running tests:
class VerifyDiffpack(Verify):
def __init__(self, root=’.’, task=’verify’, diffsummary = ’verify_log’, diffprog = ’diff.pl’, makemode = ’opt’):
# optimized or non-optimized compilation?
self.makemode = makemode
Verify.__init__(self, root, task, diffsummary) def run(self, script):
# go to parent directory (os.pardir is ’..’):
thisdir = os.getcwd(); os.chdir(os.pardir) if os.path.isfile(’Makefile’):
# Diffpack compilation command:
cmd = ’Make MODE=%s’ % self.makemode
failure, output = commands.getstatusoutput(cmd) os.chdir(thisdir) # back to regression test directory f, o = commands.getstatusoutput(script) # run test if failure: print ’Could not run regression test’, script
Theself._update method simply copies new*.vfiles to reference results in
*.r:
def _update(self, dirname, basename):
vfile = basename + ’.v’; rfile = basename + ’.r’
if os.path.isfile(vfile):
os.rename(vfile, rfile)
The final function we need to explain is the recursive walk through all sub- directories ofroot, where self._singlefile must be called for each file in a directory6:
def _search4verify(self, task, dirname, files):
"""Called by walk."""
# change directory to current directory:
origdir = os.getcwd(); os.chdir(dirname) for file in files:
self._singlefile(dirname, task, file) self.clean(dirname)
# recursive walks often get confused unless we do chdir back:
os.chdir(origdir)
The call toself.cleanis meant to clean up the directory after the regression test is performed. When running regression tests on interpreted programs
6 See page 123 for careful change of directories during anos.path.walk.
B.4. Verification of Scripts 719 (like scripts) this will normally be an empty function, whereas in subclasses likeVerifyDiffpackwe can redefinecleanto remove files from a compilation:
def clean(self, dirname):
# go to parent directory and clean application:
thisdir = os.getcwd(); os.chdir(os.pardir) if os.path.isfile(’Makefile’):
commands.getstatusoutput(’Make clean’) os.chdir(thisdir)