FERMam: a lightweight dual-source and multi-scale fusion framework for facial expression recognition

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Facial Expression Recognition (FER) is crucial for human-computer interaction but faces challenges in balancing performance and efficiency.
• Convolutional Neural Networks (CNNs) struggle with global dependencies, while Transformers have high computational complexity.
• Resource-constrained environments limit the deployment of conventional FER methods.

Purpose of the Study:

• To develop a lightweight and efficient framework for Facial Expression Recognition (FER).
• To address the limitations of existing CNN and Transformer-based FER methods in terms of performance and computational cost.
• To enable effective FER in resource-limited settings.

Main Methods:

• Proposed FERMam framework integrating dual-source and multi-scale features.
• Utilized an image fusion encoder combining CNN and Mamba-based selective state-space modeling for local and global feature extraction.
• Incorporated a facial landmark branch for geometry-aware representation and a pyramid fusion structure with an Adaptive State-space Feature Refinement (ASFR) module.

Main Results:

• FERMam demonstrated significant reductions in parameters (e.g., 62.81M fewer than POSTER) and FLOPs (e.g., 9.73G fewer than POSTER).
• Achieved comparable accuracy to existing methods on RAF-DB, AffectNet, and FERPlus datasets.
• The model proved highly efficient, suitable for deployment in resource-constrained environments.

Conclusions:

• FERMam offers a compelling solution for efficient and accurate Facial Expression Recognition.
• The proposed framework effectively balances performance and computational efficiency.
• FERMam is well-suited for practical applications in intelligent human-computer interaction with limited resources.