Toward Sensor-to-Text Generation: Leveraging LLM-Based Video Annotations for Stroke Therapy Monitoring.

Area of Science:

• Rehabilitation Medicine
• Biomedical Engineering
• Artificial Intelligence

Background:

• Stroke-related impairment is a major cause of long-term disability.
• Wearable sensors are promising for scalable rehabilitation monitoring but struggle with movement classification.
• Manual annotation of sensor data is time-consuming and inconsistent.

Purpose of the Study:

• To develop a novel framework for automated activity description generation from stroke rehabilitation therapy sessions.
• To align large language model (LLM)-generated activity descriptions with accelerometer data for creating labeled training datasets.
• To explore the feasibility of generating natural language narratives directly from sensor data.

Main Methods:

• Utilized large language models (LLMs) to generate activity descriptions from video frames of therapy sessions.
• Aligned LLM-generated descriptions with concurrently recorded accelerometer signals.
• Conducted exploratory analysis of accelerometer signals to identify distinct temporal and statistical patterns for specific activities.

Main Results:

• Demonstrated that accelerometer signals contain distinct patterns corresponding to specific activities.
• Showcased the feasibility of generating natural language narratives directly from sensor data.
• Successfully created labeled training data by aligning LLM descriptions with accelerometer signals.

Conclusions:

• The proposed framework enables automated, non-intrusive, and scalable stroke rehabilitation monitoring.
• Findings support the development of sensor-to-text models for future rehabilitation applications.
• Eliminates the need for manual or video-based annotation in stroke rehabilitation monitoring.