Standardised Versioning of Datasets: a FAIR-compliant Proposal.

Area of Science:

• Data Science
• Machine Learning
• Software Engineering

Background:

• Effective dataset versioning is crucial for reproducibility and workflow integration.
• Existing methods lack standardized tracking for data evolution and usability.
• Data drift quantification is needed to assess statistical property changes over time.

Purpose of the Study:

• To introduce a standardized dataset versioning framework.
• To develop and evaluate data drift metrics for quantifying dataset changes.
• To identify the most effective metric for tracking dataset updates and ensuring data usability.

Main Methods:

• Implementation of a "major.minor.patch" versioning nomenclature.
• Development of three data drift metrics (d_P, d_E,PCA, d_E,AE) using unsupervised Machine Learning (PCA, Autoencoders).
• Evaluation of metrics for dataset creation, update, and deletion scenarios.

Main Results:

• The d_E,PCA metric, combining PCA and splines, demonstrated efficient computation.
• Low metric values (<50) indicated new dataset batches or seasonal variations.
• High metric values (100) signified major updates (e.g., scaling >30% of variables).
• The metric showed robustness against information loss, with values near 0 for minor changes.

Conclusions:

• The proposed framework enhances dataset reusability, recognition, and version tracking.
• The d_E,PCA metric offers a favorable balance of interpretability, robustness, and computational efficiency.
• This approach facilitates informed decision-making regarding data usability and workflow integration.