M³ENet: A Multi-Modal Fusion Network for Efficient Micro-Expression Recognition.

Area of Science:

• Computer Science
• Artificial Intelligence
• Biomedical Engineering

Background:

• Micro-expression recognition (MER) is crucial for understanding suppressed emotions but faces challenges like short duration and limited data.
• Existing MER methods often rely on single cues (appearance or motion), hindering comprehensive emotional expression capture.
• Accurate MER is vital for applications like user experience analysis and security surveillance.

Purpose of the Study:

• To develop a lightweight, efficient multi-modal network for micro-expression recognition (MER).
• To effectively integrate both motion and appearance cues for enhanced MER performance.
• To improve MER generalization in low-data scenarios and address class imbalance.

Main Methods:

• Proposed M³ENet, a lightweight multi-modal fusion network integrating optical flow (motion) and RGB (appearance) features.
• Employed early-stage feature fusion within two modality-specific subnetworks.
• Utilized targeted data augmentation and focal loss for improved generalization and class imbalance mitigation.

Main Results:

• M³ENet achieved state-of-the-art performance across five benchmark datasets (CASME I, CASME II, CAS(ME)², SAMM, MMEW).
• The network demonstrated high efficiency, outperforming existing methods.
• Ablation studies and Grad-CAM visualizations validated the model's effectiveness and interpretability.

Conclusions:

• The proposed M³ENet effectively integrates multi-modal features for robust and efficient micro-expression recognition.
• The approach shows significant promise for real-world MER applications, especially in data-scarce environments.
• The study highlights the importance of multi-modal fusion and targeted training strategies for advancing MER technology.