Robust Occupant Behavior Recognition via Multimodal Sequence Modeling: A Comparative Study for In-Vehicle Monitoring...

Area of Science:

• Computer Science
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Understanding occupant behavior is crucial for intelligent transportation systems (ITS) safety.
• Multimodal data fusion enhances situational awareness in vehicles.
• Current ITS systems require robust occupant monitoring.

Purpose of the Study:

• To compare static, recurrent, and attention-based models for multimodal occupant behavior recognition.
• To evaluate model performance on a large-scale dataset.
• To identify the most effective architecture for in-vehicle monitoring.

Main Methods:

• Utilized sequential inputs from 2D pose, 2D gaze, and facial movements.
• Compared Multi-Layer Perceptron (MLP), Long Short-Term Memory (LSTM), and Transformer encoder architectures.
• Conducted experiments on the Occupant Behavior Classification (OBC) dataset (2.1M frames, 79 classes).

Main Results:

• Temporal models (LSTM, Transformer) significantly outperformed the static MLP baseline.
• The Transformer encoder achieved state-of-the-art Macro F1 score of 0.9570.
• Transformer demonstrated a strong balance between performance and computational efficiency.

Conclusions:

• Attention-based temporal modeling with multimodal fusion is superior for occupant behavior recognition.
• The Transformer architecture provides a practical framework for efficient in-vehicle monitoring.
• Findings advance the development of safer and more aware intelligent transportation systems.