Avoiding Biased Clinical Machine Learning Model Performance Estimates in the Presence of Label Selection

Area of Science:

• Machine Learning
• Clinical Informatics
• Causal Inference

Background:

• Evaluating clinical machine learning models requires understanding the deployment population.
• Label selection, where observed patients are a subset of the deployment population, can lead to misleading performance estimates.
• Standard metrics may not accurately reflect real-world performance due to biased label selection.

Purpose of the Study:

• To describe classes of label selection and simulate scenarios to assess bias in machine learning performance metrics.
• To investigate how label selection mechanisms affect model discrimination and calibration.
• To propose and evaluate methods for obtaining accurate performance estimates in deployed clinical models.

Main Methods:

• Simulated five causally distinct scenarios of label selection.
• Assessed bias in commonly reported binary machine learning performance metrics.
• Applied traditional weighting estimators from causal inference.
• Trained machine learning models to flag low-yield laboratory diagnostics.
• Proposed an altered deployment procedure combining randomization and weighted estimates.

Main Results:

• Selection affected by observed features can mislead discrimination estimates.
• Selection affected by labels can mislead calibration estimates.
• Weighting estimators, when properly specified, recover full population estimates.
• Naive AUROC estimates on the observed population undershot actual performance by up to 20% in a real-world task.
• The proposed deployment procedure recovered true model performance.

Conclusions:

• Label selection poses a significant challenge to accurate clinical machine learning model evaluation.
• Causal inference methods, particularly weighting estimators, are crucial for correcting performance bias.
• Misleading performance estimates can lead to premature termination of beneficial clinical tools.
• An altered deployment strategy combining randomization and weighting is effective in recovering true model performance.