CDGT: Constructing diverse graph transformers for emotion recognition from facial videos

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Facial expression recognition from dynamic videos is challenging due to pose variations, occlusions, and subtle emotional changes.
• Existing transformer methods struggle to focus on localized spatial-temporal features crucial for in-the-wild expression analysis.

Purpose of the Study:

• To propose a novel method, diverse graph transformers (CDGT), for efficient and accurate emotion recognition from in-the-wild facial videos.
• To enhance the modeling of localized expression-related structures in both spatial and temporal domains.

Main Methods:

• The proposed CDGT method incorporates diverse graph structures into transformer architectures.
• It features a spatial dual-graphs transformer with dual-graph constrained attention for local spatial token analysis.
• It also includes a temporal hyperbolic-graph transformer using hyperbolic-graph constrained self-attention for subtle dynamic emotion modeling.

Main Results:

• The spatial dual-graphs transformer effectively handles pose variations and partial occlusions.
• The temporal hyperbolic-graph transformer captures subtle dynamic emotional changes.
• Extensive experiments demonstrate CDGT's superior performance over state-of-the-art methods on in-the-wild video datasets.

Conclusions:

• The CDGT method offers a significant advancement in recognizing facial expressions from complex, real-world videos.
• Integrating diverse graph structures into transformers enhances the ability to capture crucial localized spatial-temporal features for emotion recognition.