Participant-Aware Model Validation for Repeated-Measures Data: Comparative Cross-Validation Study.

Area of Science:

• Biomechanics
• Digital Health
• Machine Learning

Background:

• Repeated-measures data are prevalent in biomechanics and digital health.
• Standard cross-validation (CV) can leak information between training and testing sets in such data.
• This leakage inflates performance estimates and reduces clinical credibility.

Purpose of the Study:

• To evaluate participant-aware validation strategies for repeated-measures classification tasks.
• To assess their impact on model reliability using fear of reinjury prediction after anterior cruciate ligament reconstruction (ACLR) as a case study.

Main Methods:

• Compared four CV strategies: stratified 10-fold CV, leave-one-participant-out CV (LOPOCV), group 3-fold CV, and a nested LOPOCV + group CV framework.
• Analyzed 623 hop trials from 72 individuals post-ACLR to predict fear of reinjury using biomechanical features.
• Benchmarked ten supervised classifiers on accuracy, generalization gap, ranking consistency, and computational efficiency.

Main Results:

• Stratified 10-fold CV significantly overestimated model performance due to participant data leakage.
• Group and nested CV strategies provided more conservative and stable performance estimates.
• The nested LOPOCV + group CV framework balanced generalization and participant-aware separation, reducing bias and overfitting.

Conclusions:

• Participant-aware validation is essential for trustworthy machine learning (ML) in repeated-measures settings.
• Nested CV designs enhance reproducibility and reduce bias, aligning with clinical ML regulatory expectations.
• These findings advocate for robust model validation in biomechanics and digital health.