A workflow reproducibility scale for automatic validation of biological interpretation results

Area of Science:

• Bioinformatics
• Computational Biology
• Data Science

Background:

• Reproducibility of data analysis workflows is critical in bioinformatics.
• While virtualization aids workflow execution, evaluating the biological interpretation of results remains a challenge.
• A standard method for assessing the reproducibility of biological interpretations is lacking.

Purpose of the Study:

• To propose a novel metric for evaluating the reproducibility of workflow execution results.
• To develop a system for the automatic evaluation of result reproducibility.
• To address the challenge of assessing the biological interpretation consistency in reproduced results.

Main Methods:

• Introduced a reproducibility scale based on biological feature values (e.g., read counts, mapping rates, variant frequencies).
• Implemented a prototype system for automated reproducibility evaluation.
• Experimented using real-world bioinformatics research workflows and common use cases.

Main Results:

• Developed a metric to evaluate the reproducibility of workflow results based on biological interpretation.
• Created a system capable of automatically assessing result reproducibility.
• Demonstrated the approach's effectiveness on practical bioinformatics workflows.

Conclusions:

• The proposed approach enables automatic, fine-grained evaluation of result reproducibility.
• Facilitates a move beyond binary assessments to a graduated understanding of reproducibility.
• Aims to enhance discussions and standards for reproducibility in bioinformatics research.