An Integrated Computer Vision and Force Sensing Framework for Automated Fugl-Meyer Hand-Related Assessment Using...

The Fugl-Meyer Assessment (FMA) is widely used to evaluate motor function after stroke in clinical settings; however, its reliance on therapists' visual judgment without objective measurement tools may lead to variable results due to subjectivity. To address this challenge, a portable multi-camera system was developed and integrated with computer vision techniques to track hand position and joint angles during task-related motions. A force-sensing system was additionally incorporated into the assessment tool to measure grasp strength. Data were collected from six healthy subjects (HS), each performing FMA assessment tasks designed to emulate a broad spectrum of motor abilities corresponding to FMA scores of 0, 1, and 2. These data were subsequently used to pre-train an artificial neural network (ANN) for automated FMA scoring, with eight different model architectures evaluated to characterize their performance and identify the optimal approach for this study. The resulting pre-trained model was then tested using data from six stroke patients (SP), whose motor abilities were clinically assessed by a therapist. Among the eight different optimized ANN models pre-trained exclusively on HS data, the top-performing model achieved an average accuracy of 98% across seven hand tests on the HS dataset and maintained 85% accuracy on SP data without additional fine-tuning, indicating effective knowledge transfer. Detailed evaluations revealed that autoencoder-based feature extraction improved accuracy, underscoring the importance of identifying and leveraging key features. Moreover, Long Short-Term Memory (LSTM)-based methods effectively captured the temporal dynamics of human motion data. These findings confirm that the developed computer vision and force-sensing system effectively captures hand motions and grasp strength. Moreover, the results establish a critical foundation for verifying key assumptions that learned representations from HS data, through pre-trained and optimized ANN structures, can be transferred to clinical SP data-mitigating common challenges associated with collecting patient data in clinical studies. This verification paves the way for future work, such as exploring additional fine-tuning strategies to enhance accuracy, applying advanced data augmentation techniques, incorporating larger patient datasets, and employing transfer learning for continuous scoring rather than discrete values for more refined assessments.