MOL: Joint Estimation of Micro-Expression, Optical Flow, and Landmark via Transformer-Graph-Style Convolution

Area of Science:

• Computer Vision
• Machine Learning
• Affective Computing

Background:

• Facial micro-expression recognition (MER) is challenging due to subtle, transient actions.
• Existing MER methods often rely on handcrafted features or key frames, and are limited by small datasets.
• Deep learning approaches for MER face limitations due to dataset scale and diversity.

Purpose of the Study:

• To propose an end-to-end micro-action-aware deep learning framework for MER.
• To develop a novel feature extraction block (F5C) that directly processes raw frames.
• To jointly train MER with optical flow estimation and facial landmark detection to enhance subtle action capture.

Main Methods:

• Developed a novel F5C block combining fully-connected convolution and channel correspondence convolution for local-global feature extraction.
• Utilized transformer-style and graph-style convolutions to extract local features and model feature correlations.
• Implemented joint training of MER, optical flow estimation, and facial landmark detection using shared features.

Main Results:

• The proposed framework significantly outperforms state-of-the-art MER methods on CASME II, SAMM, and SMIC benchmarks.
• The model demonstrates strong performance in optical flow estimation and facial landmark detection.
• The framework successfully captures subtle facial muscle actions associated with micro-expressions.

Conclusions:

• The proposed micro-action-aware deep learning framework offers a robust solution for facial micro-expression recognition.
• Joint training with auxiliary tasks effectively addresses data scarcity issues in MER.
• The novel F5C block enables direct, key-frame-free feature extraction from raw video sequences.