Detecting the impact of subject characteristics on machine learning-based diagnostic applications

Area of Science:

• Digital Health
• Machine Learning
• Biostatistics

Background:

• High-dimensional, longitudinal digital health data offer significant research and clinical potential.
• Developing diagnostic algorithms requires careful consideration of repeated measurements from individuals.
• Current analytical evaluations of predictive performance often overlook the impact of repeated measures.

Purpose of the Study:

• To present a method for quantifying "identity confounding" in digital health classifiers.
• To demonstrate the prevalence and impact of identity confounding in real-world digital health datasets.
• To advocate for the avoidance of record-wise data splits in machine learning applications for digital health.

Main Methods:

• Development of a novel method to calculate identity confounding.
• Application of the method to multiple real-world digital health datasets.
• Evaluation of predictive performance in machine learning models trained with different data splitting strategies.

Main Results:

• Record-wise data splits, where repeated measures from individuals are present in both training and testing sets, lead to massive underestimation of prediction error.
• Identity confounding, where models learn to identify subjects rather than diagnostic signals, is a significant issue in digital health studies.
• The proposed method effectively quantifies the extent of identity confounding present in classifiers.

Conclusions:

• Identity confounding is a critical problem in digital health research, biasing predictive performance evaluations.
• Record-wise data splits are inappropriate for developing and evaluating machine learning-based digital health applications.
• Future research must adopt data splitting strategies that mitigate identity confounding to ensure reliable diagnostic algorithms.