The need to approximate the use-case in clinical machine learning.

Area of Science:

• Digital Health
• Machine Learning in Medicine
• Biomedical Data Science

Background:

• Smartphone and wearable sensors generate vast human subject data.
• Machine learning (ML) models are increasingly used for clinical predictions.
• Accurate evaluation of ML model performance is critical for clinical decision-making.

Purpose of the Study:

• To compare the accuracy of record-wise versus subject-wise cross-validation (CV) for ML models using sensor data.
• To assess the clinical relevance of different CV strategies.
• To identify the prevalence of suboptimal CV methods in the literature.

Main Methods:

• Comparison of record-wise and subject-wise CV using a public dataset and simulations.
• Systematic literature review of studies employing wearable sensors for clinical outcome prediction.
• Analysis of CV methods used in retrieved studies.

Main Results:

• Record-wise CV frequently overestimates ML model prediction accuracy.
• Subject-wise CV better reflects real-world clinical scenarios (e.g., new patient diagnosis).
• Nearly half of reviewed studies used the less reliable record-wise CV method.

Conclusions:

• Record-wise CV can lead to misleadingly optimistic accuracy estimates for ML models.
• Subject-wise CV is crucial for reliable evaluation of ML models in clinical applications.
• Adoption of appropriate CV methods is essential for trustworthy ML-driven healthcare.