External validation of machine learning models-registered models and adaptive sample splitting.

Area of Science:

• Biomedical research
• Translational medicine
• Machine learning

Background:

• Multivariate predictive models are vital for understanding biological systems and developing translational medical research tools.
• Model complexity and extensive data preprocessing can lead to overfitting and poor generalizability.
• External validation on independent data is crucial for unbiased evaluation but often neglected due to costs.

Purpose of the Study:

• To propose methods for enhancing the credibility of predictive models in translational research.
• To introduce a novel approach for optimizing the balance between model discovery and external validation efforts.
• To address issues of replicability, effect size inflation, and generalizability in predictive modeling.

Main Methods:

• Separating model discovery and external validation through public disclosure (e.g., preregistration) of feature processing and model weights.
• Implementing a novel adaptive splitting approach to optimize the trade-off between model discovery and external validation.
• Testing the approach on over 3,000 participants across four independent datasets.

Main Results:

• The proposed method ensures maximal credibility by separating model discovery and external validation.
• The adaptive splitting approach identifies the optimal time to cease model discovery for maximized predictive performance.
• This approach prevents risking low-powered and inconclusive external validation.

Conclusions:

• The proposed design and splitting approach, implemented in the "AdaptiveSplit" Python package, can improve replicability.
• The method helps mitigate effect size inflation in predictive modeling studies.
• It contributes to enhancing the generalizability of predictive models in medical research.