A Vision-based System for Monitoring Eating Behaviors and Musculoskeletal Function

Camera-based systems offer a comprehensive and inconspicuous approach to monitoring the well-being of individuals within the comfort of their homes. This study introduces a vision-based, fully autonomous pipeline for assessing eating behaviors and detecting musculoskeletal changes. The system captures eating activities and provides detailed insights such as hand-to-mouth motion duration and bite count. These indicators are vital for understanding behavioral and physiological influences on food consumption and their associated changes. The system integrates pose estimation and a temporal action localization network to classify actions and generate behavior profiles. Evaluated on the EatSense dataset and a supplementary test set, the system achieves strong performance, including a mean average precision (mAP) of 74% at 0.10 IoU for micro-action detection and a posture anomaly detection accuracy of over 76%. These results demonstrate the system's ability to detect subtle trends such as slower hand movements under increased wrist weights and changes in chewing behavior. Additionally, comparisons against Gemini-2.5-Pro, a state-of-the-art multimodal model, reinforces the system's accuracy. So, by successfully capturing trends aligned with ground truth data, the pipeline shows promise for long-term health monitoring, early detection of musculoskeletal decline, and behavioral changes in dietary habits-offering potential applications in elderly care and remote health assessment. The new test dataset is released on https://groups.inf.ed.ac.uk/vision/DATASETS/EATSENSE/.