Exploratory Metamorphic Testing for Scientific Software

Scientific model developers are able to verify and validate their software via metamorphic testing, even when the expected output of a given test case is not readily available. The tenet is to check whether certain relations hold among the expected outputs of multiple related inputs. Contemporary approaches require the relations to be defined before tests. Our experience shows that it is often straightforward to first define the multiple iterations of tests for performing continuous simulations, and then keep multiple and even competing metamorphic relations open for investigating the testing-result patterns. We call this new approach exploratory metamorphic testing, and report our experience of applying it to detect bugs, mismatches, and constraints in automatically calibrating parameters for the United States Environmental Protection Agency's Storm Water Management Model (SWMM).