Related Experiment Video
Updated: Apr 25, 2026

Comparing Bibliometric Analysis Using PubMed, Scopus, and Web of Science Databases
Published on: October 24, 2019
Testing Scientific Software: A Systematic Literature Review
Upulee Kanewala1, James M Bieman1
1Computer Science Department, Colorado State University, USA.
Context:
Scientific software plays an important role in critical decision making, for example making weather predictions based on climate models, and computation of evidence for research publications. Recently, scientists have had to retract publications due to errors caused by software faults. Systematic testing can identify such faults in code.
Objective:
This study aims to identify specific challenges, proposed solutions, and unsolved problems faced when testing scientific software.
Method:
We conducted a systematic literature survey to identify and analyze relevant literature. We identified 62 studies that provided relevant information about testing scientific software.
Results:
We found that challenges faced when testing scientific software fall into two main categories: (1) testing challenges that occur due to characteristics of scientific software such as oracle problems and (2) testing challenges that occur due to cultural differences between scientists and the software engineering community such as viewing the code and the model that it implements as inseparable entities. In addition, we identified methods to potentially overcome these challenges and their limitations. Finally we describe unsolved challenges and how software engineering researchers and practitioners can help to overcome them.
Conclusions:
Scientific software presents special challenges for testing. Specifically, cultural differences between scientist developers and software engineers, along with the characteristics of the scientific software make testing more difficult. Existing techniques such as code clone detection can help to improve the testing process. Software engineers should consider special challenges posed by scientific software such as oracle problems when developing testing techniques.
Related Concept Videos
Statistical Software for Data Analysis and Clinical Trials
The Scientific Method
Generally, predictions are tested using carefully-designed experiments. Based on the outcome of these...
The Scientific Method in Nursing Process
When using research findings to change practice, one must understand the process used to guide a study. The scientific method is a systematic, step-by-step process that supports the data's validity, reliability, and generalizability. As a result, findings can be...
Statistical Hypothesis Testing
Statistical significance measures the probability that an observed result occurred by chance. If this probability, known as...
Systematic Error: Methodological and Sampling Errors
Sampling errors originate from improper sampling methods or the wrong sample population. These errors can be minimized by refining the sampling strategy. Defective instruments or faulty calibrations are the sources of instrumental...
Quality Assurance
