Vision-skeleton dual-modality framework for generalizable assessment of Parkinson's disease gait

Gait abnormalities in Parkinson's disease (PD) can reflect the extent of dysfunction, and making their assessment crucial for the diagnosis and treatment of PD. Current video-based methods of PD gait assessment are limited to only focusing on skeleton motion information and are confined to evaluations from a single perspective. To overcome these limitations, we propose a novel vision-skeleton dual-modality framework, which integrates keypoints vision features with skeleton motion information to enable a more accurate and comprehensive assessment of PD gait. We firstly introduce the Keypoints Vision Transformer, a novel architecture designed to extract vision features of human keypoints. This model encompasses both the spatial locations and connectivity relationships of human keypoints. Subsequently, through the proposed temporal fusion encoder, we integrate the extracted skeleton motion with keypoints vision features to enhance the extraction of temporal motion features. In a video dataset of 241 PD participants recorded from the front, our proposed framework achieves an assessment accuracy of 78.05%, which demonstrates superior performance compared to other methods. To enhance the interpretability of our method, we also conduct a feature visualization analysis of the proposed dual-modality framework, which reveal the mechanisms of different body parts and dual-modality branch in PD gait assessment. Additionally, when applied to another video dataset recorded from a more general perspective, our method still achieves a commendable accuracy of 73.07%. This achievement demonstrates the robust generalization capability of the proposed model in PD gait assessment from cross-view, which offers a novel approach for realizing unrestricted PD gait assessment in home monitoring. The latest version of the code is available at https://github.com/FJNU-LWP/PD-gait-VSDF.