Learning Facial Action Units with Spatiotemporal Cues and Multi-label Sampling.

Area of Science:

• Computer Vision
• Machine Learning
• Human-Computer Interaction

Background:

• Facial action units (AUs) are crucial for understanding facial expressions.
• Previous research often analyzes AUs spatially or temporally, but not jointly.
• Existing methods may exhibit person-specific biases and struggle with sparse AU data.

Purpose of the Study:

• To develop a hybrid network architecture for joint spatial and temporal AU representation modeling.
• To improve the accuracy and reduce biases in AU detection.
• To address class imbalance issues in AU datasets.

Main Methods:

• A hybrid network combining Convolutional Neural Networks (CNNs) for spatial features and Long Short-Term Memory (LSTM) networks for temporal dependencies.
• A fusion network aggregates CNN and LSTM outputs for per-frame AU prediction.
• Introduction of multi-labeling sampling strategies to handle class imbalance.

Main Results:

• The hybrid system demonstrated reduced person-specific biases compared to state-of-the-art methods.
• Increased accuracy in AU detection was achieved on the GFT and BP4D datasets.
• Multi-labeling sampling strategies further enhanced accuracy, particularly for sparse AUs.

Conclusions:

• Jointly modeling spatial, temporal, and correlational aspects of AUs leads to superior detection performance.
• The proposed hybrid network offers a more robust and accurate approach to facial action unit recognition.
• Visualizations provide novel insights into machine perception of facial actions.