Reducing annotation burden in physical activity research using vision language models.

Area of Science:

• Wearable sensor technology
• Computer vision
• Machine learning for health

Background:

• Wearable device data is crucial for validating measurement approaches and developing new machine learning models for health research.
• Human annotation of body-worn camera data is a common but labor-intensive method for labeling physical activity behaviors.

Purpose of the Study:

• To investigate the accuracy of open-source vision-language models (VLMs) in annotating physical activity intensity from wearable camera images.
• To assess whether VLMs can reduce the annotation burden in wearable camera-based validation studies.

Main Methods:

• Compared three VLMs and two discriminative models (DMs) using data from two free-living studies (Oxfordshire, UK; Sichuan, China).
• Utilized data from the Autographer wearable camera across studies with 161 and 111 participants.
• Evaluated model performance on predicting activity intensity classes (sedentary, light, moderate-to-vigorous physical activity).

Main Results:

• The best VLM and a fine-tuned DM showed comparable performance for sedentary behavior in the Oxfordshire study (median F1-scores: VLM=0.89, DM=0.91).
• Performance decreased for light and moderate-to-vigorous physical activity for both model types.
• Model performance significantly declined when applied to the external Sichuan study, indicating challenges with generalizability.

Conclusions:

• Freely available computer vision models show potential for annotating sedentary behavior from wearable camera images within similar populations.
• These models can help reduce the annotation burden for ground-truth data collection in physical activity research.
• Further research is needed to improve model generalizability across diverse populations and activity types.