Loss function influence on hyperparameter optimization for observational healthcare prediction models

Area of Science:

• Healthcare data science
• Clinical prediction modeling
• Machine learning in medicine

Background:

• Machine learning models are vital for healthcare risk stratification and personalized care.
• Hyperparameter tuning is essential for optimizing model performance using specific loss function metrics.
• The area under the receiver operating characteristic curve (AUROC) is a common, but not always optimal, metric for healthcare applications.

Purpose of the Study:

• To empirically assess how the choice of loss function metric impacts hyperparameter optimization and model behavior in clinical prediction tasks.
• To investigate systematic differences in model performance and individual predictions based on different loss function metrics.
• To evaluate the effect of loss function choice on real-world healthcare data.

Main Methods:

• Utilized fifteen distinct loss function metrics for hyperparameter selection.
• Applied four machine learning algorithms across three clinical prediction tasks.
• Compared the influence of loss function choice on hyperparameters, overall performance (AUROC, discrimination, calibration), and individual predicted probabilities.

Main Results:

• Hyperparameter optimal values varied by algorithm but showed some consistency across loss functions.
• Models optimized using AUROC were not always the top performers when evaluated by AUROC.
• Population-level model performance (discrimination, calibration) was similar across loss functions, but individual patient risk predictions varied significantly.
• The choice of loss function metric substantially impacted individual predicted patient risk.

Conclusions:

• Significant "predictive multiplicity" occurs at the patient level due to loss function choice, despite similar population-level performance.
• This patient-level variability can critically influence treatment decisions and requires further understanding.
• The study highlights the importance of carefully selecting loss function metrics in clinical machine learning to ensure accurate and equitable patient care.