Data splitting to avoid information leakage with DataSAIL

Area of Science:

• Machine Learning
• Bioinformatics
• Computational Biology

Background:

• Information leakage during training can cause machine learning models to memorize data, leading to overestimated performance.
• Accurate evaluation of machine learning models is crucial for reliable deployment in biomedical applications, especially in out-of-distribution scenarios.

Purpose of the Study:

• To introduce DataSAIL, a Python package designed to mitigate information leakage in data splitting for machine learning models.
• To enable more realistic performance evaluations of biomedical machine learning models by reducing data leakage.

Main Methods:

• Formulating the problem of leakage-reduced data splitting as a combinatorial optimization problem.
• Proving the NP-hard nature of the problem and developing a scalable heuristic solution using clustering and integer linear programming.
• Implementing the heuristic within the DataSAIL Python package.

Main Results:

• DataSAIL provides a method for generating data splits that minimize information leakage.
• The package facilitates more reliable assessment of machine learning model generalizability in biomedical contexts.
• Empirical results demonstrate the positive impact of DataSAIL on evaluating machine learning models for biological data.

Conclusions:

• DataSAIL offers a practical solution for addressing information leakage in machine learning for biomedical applications.
• The package promotes robust model evaluation, crucial for trustworthy AI in healthcare and life sciences.
• By enabling leakage-reduced data splitting, DataSAIL supports the development of more generalizable and reliable biomedical AI tools.