A framework for understanding label leakage in machine learning for health care

Area of Science:

• Clinical prediction modeling
• Machine learning in healthcare

Background:

• Label leakage, where outcome information contaminates model features, is a known pitfall in developing clinical prediction models.
• Standard advice like the "no time machine rule" is insufficient for navigating the nuances of avoiding label leakage in complex clinical settings.

Purpose of the Study:

• To provide a comprehensive framework for assessing potential label leakage in clinical prediction models.
• To offer practical guidance for stakeholders on evaluating leakage tradeoffs during model development and implementation.

Main Methods:

• Development of a framework based on prediction cadence, perspective, and applicability.
• Analysis of real-world clinical models to illustrate appropriate and inappropriate label leakage.

Main Results:

• The study highlights that avoiding label leakage requires a nuanced approach beyond simple temporal rules.
• Real-world examples demonstrate the practical challenges and solutions in managing label leakage.

Conclusions:

• Implementing a structured framework aids in identifying and mitigating label leakage risks.
• Standardized language and evaluation dimensions can improve the development of statistically valid and clinically useful machine learning models.