SciDataFlow: a tool for improving the flow of data through science.

Area of Science:

• Computational science
• Data management
• Scientific reproducibility

Background:

• Managing large datasets alongside code in platforms like GitHub is challenging.
• Iterative data analysis risks unnoticed data changes, compromising reproducibility.

Purpose of the Study:

• To introduce SciDataFlow, a tool designed to streamline data and code management in open scientific research.
• To address the challenges of large dataset storage and tracking data modifications in computational analyses.

Main Methods:

• Developed SciDataFlow as a fast, concurrent command-line tool.
• Implemented a simple Data Manifest specification for tracking data.
• Enabled streamlined uploading of data to remote repositories.
• Facilitated pulling necessary data for reproducible analyses.

Main Results:

• SciDataFlow effectively tracks data changes, preventing the use of outdated data versions.
• The tool simplifies uploading and retrieving data for computational analyses.
• Ensures all data required for reproducing an analysis is readily available.

Conclusions:

• SciDataFlow enhances data integrity and reproducibility in open scientific research.
• The tool offers a practical solution for managing large datasets and code in repositories.
• SciDataFlow promotes more reliable and transparent scientific workflows.