Confound-leakage: confound removal in machine learning leads to leakage

Area of Science:

• Computational Biology
• Epidemiology
• Medical Informatics

Background:

• Machine learning (ML) is vital for data analysis in epidemiology and medicine, with nonlinear methods excelling at complex predictions.
• ML models can be biased by confounding information in features.
• Featurewise linear confound regression (CR) is a standard method to address confounding, but its pitfalls in ML are not fully understood.

Purpose of the Study:

• To investigate the potential risks of using linear confound regression (CR) within machine learning (ML) pipelines.
• To demonstrate how CR can inadvertently amplify confounding effects, leading to unreliable predictions.

Main Methods:

• Developed a simple framework using the target variable as a confound to analyze CR's impact.
• Employed feature shuffling to distinguish between confound-leakage and genuine information revelation.
• Evaluated the method in a real-world clinical prediction task for attention-deficit/hyperactivity disorder.

Main Results:

• Linear CR can increase confounding risk when combined with nonlinear ML, contrary to expectations.
• Information leakage via CR can inflate effect sizes, leading to overestimated predictive accuracy.
• Demonstrated overestimation in predicting ADHD using speech features when depression was used as a confound.

Conclusions:

• Improper use of CR can lead to untrustworthy, biased, and unfair ML predictions due to confound-leakage.
• Understanding and addressing the confound-leakage pitfall is crucial for developing robust and reliable ML models.
• Guidelines are provided to mitigate these risks in ML applications.