Risk of bias in studies on prediction models developed using supervised machine learning techniques: systematic...

Area of Science:

• Medical Informatics
• Machine Learning in Healthcare
• Clinical Prediction Models

Background:

• Machine learning (ML) is increasingly used for developing diagnostic and prognostic prediction models in medicine.
• Assessing the methodological quality and risk of bias in these studies is essential for reliable clinical application.

Purpose of the Study:

• To systematically review and assess the methodological quality and risk of bias of studies developing prediction models using machine learning techniques across all medical specialties.

Main Methods:

• A systematic review of studies published between January 2018 and December 2019 was conducted.
• The Prediction Risk Of Bias Assessment Tool (PROBAST) was used to evaluate methodological quality and risk of bias in 152 developed models and 19 external validations.
• Risk of bias was assessed across domains: participants, predictors, outcome, and analysis.

Main Results:

• Out of 171 analyses, 148 (87%) were rated at high risk of bias, with the analysis domain most frequently affected.
• Key issues included inadequate events per predictor (56%), poor handling of missing data (41%), and improper assessment of overfitting (39%).
• While data sources were generally appropriate, information on blinding was often absent.

Conclusions:

• The majority of studies developing ML-based prediction models exhibit poor methodological quality and a high risk of bias.
• Addressing issues like small study size, missing data, and overfitting is critical.
• Improving the design, conduct, reporting, and validation of these studies is necessary to facilitate their integration into clinical practice.