Multi-Granularity Facial Emotional Representation With Unlabeled Data and Textual Supervision

Area of Science:

• Computer Vision
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Facial expressions (FEs) and action units (AUs) represent emotions at different granularities.
• Recognizing FEs and AUs are typically separate tasks, with few unified models available.
• Existing methods show limited success in simultaneously recognizing both FEs and AUs.

Purpose of the Study:

• To develop a unified model for joint facial expression recognition (FER) and action unit detection (AUD).
• To enhance the generalization capability of models for facial emotional representations.
• To address the challenge of limited annotated data for both FEs and AUs.

Main Methods:

• Constructed a unified model for joint FER and AUD.
• Utilized large amounts of unlabeled facial data from the wild.
• Designed category-specific confidence margins and leveraged FE-AU correspondences for pseudo-labeling.
• Incorporated semantically richer textual descriptions as supervision, refined through visual perception.
• Leveraged correlations between AUs and between FEs and AUs for precision enhancement.

Main Results:

• Demonstrated the superiority of the proposed unified model.
• Achieved strong generalization capabilities across multiple datasets.
• Evaluated performance through a unified zero-shot benchmark, within-domain, and cross-domain evaluations.
• Showcased effective utilization of unlabeled data and textual supervision.

Conclusions:

• The proposed unified model effectively performs joint FER and AUD.
• The method exhibits strong generalization capabilities for facial emotional representation.
• Leveraging unlabeled data and textual descriptions significantly enhances model performance and robustness.