Vision-language models for human motion understanding: Lessons from stroke rehabilitation

Area of Science:

• Artificial Intelligence
• Computer Vision
• Digital Health

Background:

• Vision-language models (VLMs) excel in computer vision tasks.
• Their application in digital health, particularly stroke rehabilitation, is emerging.
• Quantifying rehabilitation dose and impairment from videos is crucial for data-driven approaches.

Purpose of the Study:

• To apply VLMs to automatically quantify stroke rehabilitation dose and impairment from videos.
• To assess the current capabilities and limitations of VLMs for these tasks.

Main Methods:

• Formulated rehabilitation dose and impairment quantification as motion-identification tasks for VLMs.
• Evaluated a VLM framework on videos from 20 healthy controls and 51 stroke survivors.
• Tested VLM performance with optimized prompting and post-processing.

Main Results:

• Current VLMs lack the fine-grained motion understanding for precise quantification; dose estimates were comparable to non-visual baselines, and impairment scores were unreliable.
• VLMs demonstrated potential by classifying high-level activities and detecting motion/grasp with moderate accuracy.
• Approximated dose counts within 30% of ground truth for healthy and mildly impaired participants without task-specific training.

Conclusions:

• VLMs currently have limitations for precise, data-driven stroke rehabilitation quantification.
• Optimized VLM application shows emerging opportunities for clinical video analysis and rehabilitation monitoring.
• Further development is needed to enhance VLM motion understanding for clinical applications.