Pitfalls in using ML to predict cognitive function performance.

Area of Science:

• Cognitive Neuroscience
• Computational Linguistics
• Psychometrics

Background:

• Machine learning (ML) is frequently applied to predict cognitive abilities.
• Implementation and interpretation of ML models carry risks, particularly from confounding variables.

Purpose of the Study:

• To illustrate the risks of erroneous conclusions from ML predictions due to confounding variables.
• To demonstrate potential confound leakage in predicting executive function (EF) using prosodic features.

Main Methods:

• Healthy participants (n=231) completed speech tasks and EF tests.
• ML models predicted EF performance using 264 prosodic features, controlling for age, sex, and education.
• In-depth analyses examined potential confound leakage in prediction accuracies.

Main Results:

• ML models initially showed reasonable prediction performance for executive function (EF) variables (Trail Making Test).
• In-depth analysis revealed inflated prediction accuracies due to significant relationships between confounds (age, sex, education) and target EF performance.
• Evidence of "confound leakage" was identified, distorting the true predictive power of prosodic features.

Conclusions:

• Confounding variables pose significant risks in ML-driven cognitive ability predictions.
• Rigorous control of confounding variables is essential in ML pipelines to avoid erroneous conclusions.
• Researchers must be cautious about potential pitfalls and interpret ML prediction results carefully.