Toward intelligent rehabilitation: Multimodal human pose modeling with parametric meshes and graph-based temporal...

Area of Science:

• Biomedical Engineering
• Computer Vision
• Rehabilitation Science

Background:

• Accurate physiotherapy exercise assessment is vital for rehabilitation, especially for the elderly and mobility-impaired.
• Telerehabilitation offers a convenient alternative to in-clinic supervision, but current methods using single sensors lack robustness.
• There is a need for reliable, adaptable systems for home-based physiotherapy exercise recognition.

Purpose of the Study:

• To develop a multimodal, markerless framework for accurate and reliable home-based physiotherapy exercise recognition.
• To enhance telerehabilitation by improving the robustness and adaptability of exercise monitoring systems.

Main Methods:

• A deep learning framework integrating synchronized RGB and depth data streams.
• Extraction of 2D keypoints, body-part labels, and visual descriptors from RGB data.
• Estimation of 3D joint positions and full-body mesh reconstruction from depth data, followed by feature fusion and classification using Graph Convolutional Networks.

Main Results:

• The framework achieved high classification accuracies on three public datasets: 95.30% (KIMORE), 92.70% (mRI), and 95.59% (UTKinect-Action3D).
• Consistent performance was demonstrated across diverse rehabilitation-oriented benchmarks.
• The multimodal approach proved effective for physiotherapy exercise recognition.

Conclusions:

• Integrating complementary RGB and depth data enhances the accuracy and robustness of home-based physiotherapy exercise recognition.
• The proposed framework shows significant potential for supporting accessible and effective telerehabilitation.
• Future work will focus on broader validation and practical deployment for real-world applications.