The path toward equal performance in medical machine learning.

Area of Science:

• Machine Learning in Healthcare
• Health Equity
• Medical Informatics

Background:

• Disparities in machine learning model performance across patient groups can compromise equitable quality of care.
• Two primary mechanisms contribute to these performance differences: suboptimal performance relative to the theoretical maximum and inherent differences in prediction task difficulty between groups.

Purpose of the Study:

• To elucidate the distinct mechanisms causing performance disparities in machine learning models across patient groups.
• To analyze the impact of group underrepresentation and task characteristics on model underperformance.
• To explore factors contributing to differing optimal achievable performance levels between patient groups.

Main Methods:

• Examination of scenarios where underrepresentation leads to underperformance versus scenarios where it does not.
• Discussion of potential causes for intrinsic differences in prediction task difficulty.
• Analysis of confounding factors like label and selection biases in model learning and evaluation.

Main Results:

• Underrepresentation, modeling choices, and task characteristics can lead to model performance worse than theoretically achievable within specific groups.
• Differences in the intrinsic difficulty of prediction tasks can result in varying optimal achievable performance levels across groups.
• Label and selection biases can significantly confound both the learning process and the evaluation of model performance.

Conclusions:

• Achieving equitable machine learning model performance necessitates addressing both data quantity and quality for underperforming groups.
• Strategies to "level up" model performance must consider the nuanced interplay of data representation, modeling decisions, and task-specific challenges.
• Mitigating biases and understanding task difficulty are crucial for advancing health equity through machine learning.