Pitfalls in using ML to predict cognitive function performance.

Area of Science:

• Cognitive Neuroscience
• Computational Linguistics
• Psychometrics

Background:

• Machine learning (ML) is frequently applied to predict cognitive abilities from various data sources.
• Potential pitfalls exist in ML implementation and result interpretation, particularly concerning confounding variables.
• Executive function (EF) performance prediction using speech features is a relevant application area.

Purpose of the Study:

• To highlight the risks of erroneous conclusions in ML predictions due to confounding variables.
• To illustrate these risks with a case example predicting EF performance using prosodic features.
• To emphasize the importance of controlling for confounders in ML pipelines.

Main Methods:

• Healthy participants (n=231) completed speech tasks and EF tests.
• ML models were used to predict EF performance from 264 prosodic features.
• Confounding effects of age, sex, and education were controlled for.

Main Results:

• A reasonable model fit was initially observed for predicting EF performance on the Trail Making Test.
• In-depth analysis revealed confound leakage, leading to inflated prediction accuracies.
• A strong relationship between confounds and targets was identified as the cause of inflated accuracy.

Conclusions:

• Confounding variables can significantly impact ML model performance and accuracy.
• Failure to adequately control for confounders can lead to erroneous conclusions in cognitive prediction tasks.
• Researchers must exercise caution and implement robust methods to control for confounding variables in ML analyses.