Estimating individual minimum calibration for deep-learning with predictive performance recovery: An example case of...

Area of Science:

• Machine Learning
• Biomedical Engineering
• Data Science

Background:

• Clinical datasets frequently contain multiple trials from single participants, posing challenges for machine learning model training.
• Standard random-wise data splitting can lead to participant data appearing in both training and testing sets, compromising model generalizability.
• Subject-wise splitting segregates participant data but often results in underperforming models compared to random splits.

Purpose of the Study:

• To investigate the relationship between the size of a calibration training set and the prediction accuracy of machine learning models.
• To determine the optimal number of calibration trials needed to bridge the performance gap between subject-wise and random-wise data splits.

Main Methods:

• Development of a deep-learning classifier using a dataset of 30 healthy adults performing walking trials across nine surfaces with lower limb inertial measurement unit sensors.
• Comparison of model performance using subject-wise splits versus random-wise splits.
• Evaluation of the impact of varying calibration training set sizes on model accuracy.

Main Results:

• Subject-wise trained models showed a significant performance increase with calibration.
• A 70% increase in F1-score was observed with calibration using a single gait cycle per surface.
• 10 gait cycles per surface were sufficient for subject-wise models to achieve performance comparable to random-wise trained models.

Conclusions:

• Calibration is crucial for optimizing machine learning models trained on clinical data with subject-wise splits.
• Minimal calibration data (10 gait cycles per surface) can significantly improve model performance, making subject-wise splits more viable.
• This study provides insights into optimizing data splitting strategies for clinical machine learning applications.