Skeleton-Based Intake Gesture Detection With Spatial-Temporal Graph Convolutional Networks.

Area of Science:

• Computer Vision
• Human-Computer Interaction
• Biomedical Engineering

Background:

• Obesity and overweight are significant health issues linked to poor eating habits.
• Automated food intake gesture detection can improve dietary monitoring in daily life.
• Skeleton-based methods offer privacy and robustness for gesture recognition.

Purpose of the Study:

• To develop and validate a skeleton-based approach for automated food intake gesture detection.
• To evaluate the performance of the proposed ST-GCN-BiLSTM model on multiple datasets.
• To assess the system's potential as an automated annotation tool for nutritional analysis.

Main Methods:

• A skeleton-based approach combining a dilated spatial-temporal graph convolutional network (ST-GCN) with a bidirectional long-short-term memory (BiLSTM) framework was developed.
• The model, termed ST-GCN-BiLSTM, was trained and validated using the OREBA dataset (laboratory videos) and a self-collected smartphone dataset.
• Performance was evaluated using segmental F1-scores for eating and drinking gesture detection.

Main Results:

• The ST-GCN-BiLSTM model achieved high F1-scores on the OREBA dataset: 86.18% for eating and 74.84% for drinking gestures.
• Cross-dataset validation showed strong performance on the self-collected dataset: 85.40% for eating and 67.80% for drinking gestures.
• Results confirm the feasibility and robustness of skeleton data for intake gesture detection across varied conditions.

Conclusions:

• Skeleton-based intake gesture detection is feasible and effective for dietary monitoring.
• The proposed ST-GCN-BiLSTM model demonstrates robustness and generalizability across datasets.
• This system can serve as a valuable automated tool for nutritional experts to analyze eating behaviors.