Enhancing Facial Expression Recognition through Light Field Cameras

Area of Science:

• Computer Vision
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Facial Expression Recognition (FER) is crucial for human-computer interaction.
• Existing FER systems often rely on single modalities, limiting comprehensive analysis.
• Light field cameras offer rich data, including depth and sub-aperture information, for improved FER.

Purpose of the Study:

• To develop a more effective and comprehensive FER system by investigating multimodal fusion strategies.
• To evaluate the performance of different fusion techniques at decision and feature levels.
• To leverage complementary information from sub-aperture (SA), all-in-focus (AiF), and depth images.

Main Methods:

• Utilized EfficientNetV2-S, pre-trained on AffectNet, as the backbone convolutional neural network.
• Employed a Bidirectional Gated Recurrent Unit (BiGRU) for processing SA images.
• Investigated various decision-level and feature-level fusion strategies for multimodal data integration.

Main Results:

• The unimodal model using SA images achieved state-of-the-art performance (88.13% subject-specific, 91.88% subject-independent accuracy).
• Multimodal fusion significantly improved FER accuracy compared to unimodal approaches.
• Decision-level fusion with average weights yielded the highest accuracy (90.13% subject-specific, 93.33% subject-independent).

Conclusions:

• Multimodal fusion of SA, AiF, and depth images enhances FER system accuracy and robustness.
• The proposed approach outperforms existing FER methods.
• Decision-level fusion is a highly effective strategy for integrating complementary facial expression information.