Facial Action Unit Representation Based on Self-Supervised Learning With Ensembled Priori Constraints

Area of Science:

• Computer Vision
• Machine Learning
• Human-Computer Interaction

Background:

• Facial Action Units (AUs) are crucial for understanding human expressions.
• Supervised learning for AU recognition requires extensive manual annotations, limiting real-world performance.
• Accurate AU localization and characterization are challenging due to annotation costs.

Purpose of the Study:

• To propose an end-to-end self-supervised model (SsupAU) for learning AU representations from unlabeled facial videos.
• To overcome the limitations of manual annotations in traditional supervised AU recognition methods.
• To enable robust AU recognition in realistic scenarios.

Main Methods:

• Utilized auto-encoders to decompose input faces into six components, including photo-geometric elements and 2D flow field AUs.
• Gradually constructed canonical neutral, posed neutral, and posed expressional faces to disentangle components without supervision.
• Employed identity consistency and average face assumptions to construct the canonical neutral face and decouple AUs.

Main Results:

• Achieved superior performance in AU representation learning compared to state-of-the-art methods on benchmark datasets.
• Demonstrated an outstanding capability in decomposing input faces into meaningful factors for reconstruction.
• Successfully learned AU representations from unlabeled facial videos, validating the self-supervised approach.

Conclusions:

• The proposed SsupAU model offers an effective self-supervised method for learning facial action unit representations.
• This approach significantly reduces the reliance on manual annotations, paving the way for more practical expression understanding systems.
• The method shows promise for advancing research in facial expression analysis and human-computer interaction.