The open archive for STFC research publications

Full Record Details

Persistent URL http://purl.org/net/epubs/work/40887
Record Status Checked
Record Id 40887
Title TolDiff: A regression testing tool for validating results from scientific codes
Abstract The current practices in testing scientific and legacy codes are analysed. In particular problems with validating results in regression testing are highlighted. Commonly ad hoc solutions are implemented to deal with these problems. These solutions are typically code specific, require a non-trivial initial development effort and at least some ongoing maintenance, and even then some code errors that express themselves in the output may go undetected. To counter this unsatisfactory state of affairs an alternative approach is proposed that is based on the well known Diff program. The approach is derived from the realisation that a traditional Diff program would offer a good solution if previously identified insignificant differences could be suppressed and only potentially important differences reported. Such a tolerant diff approach requires a reference file that defines the ?gold standard? and a set of tolerances. It should support processes to diff a test result, to update the tolerances with newly discovered unimportant differences, and to transfer existing tolerances to new reference files. It is shown that such an approach can be implemented in such a way as to fully automate the diff, update, and transfer operations. It is shown that about 13 updates are needed to suppress all insignificant differences. The current implementation performs any of its operations in at worst 0.39 seconds on files of 3000 lines using a 2.8 GHz Pentium 4 machine, thus showing that practical application of this approach is feasible.
Organisation CCLRC , CSE , CSE-CCG
Keywords file differencing , regression testing , scientific computing
Funding Information
Related Research Object(s):
Licence Information:
Language English (EN)
Type Details URI(s) Local file(s) Year
Journal Article software-practice & experience (2007). 2007