Weakly Supervised Micro- and Macro-Expression Spotting Based on Multi-Level Consistency

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Micro- and macro-expression spotting in videos is challenging due to extensive data collection and frame-level annotation requirements.
• Existing weakly supervised expression spotting (WES) methods, often based on multiple instance learning (MIL), face significant inter-modality, inter-sample, and inter-task gaps.
• The inter-sample gap, particularly concerning sample distribution and duration, hinders performance in current WES approaches.

Purpose of the Study:

• To propose a novel and simplified WES framework, MC-WES, designed to achieve fine-grained frame-level spotting using only video-level labels.
• To address the limitations of existing WES methods by mitigating various inter-gap issues and integrating prior knowledge.
• To develop a system that alleviates the burden of frame-wise annotation while maintaining high spotting accuracy.

Main Methods:

• MC-WES employs multi-consistency collaborative mechanisms, including modal-level saliency, video-level distribution, label-level duration, and segment-level feature consistency strategies.
• Modal-level saliency consistency captures correlations between raw images and optical flow.
• Video-level distribution consistency leverages temporal sparsity differences; label-level duration consistency exploits facial muscle duration variations; segment-level feature consistency ensures similarity for features under identical labels.

Main Results:

• MC-WES demonstrates effective fine frame-level spotting capabilities using only video-level labels.
• The proposed multi-consistency strategies successfully alleviate identified gaps in existing WES methods.
• Experimental results on CAS(ME)$^{2}$, CAS(ME)$^{3}$, and SAMM-LV datasets show MC-WES performance comparable to state-of-the-art fully supervised methods.

Conclusions:

• MC-WES offers a simplified yet powerful approach to weakly supervised expression spotting, reducing annotation complexity.
• The multi-consistency framework effectively merges prior knowledge and addresses inherent gaps in MIL-based WES.
• MC-WES achieves competitive performance, suggesting its viability as an alternative to fully supervised methods for expression spotting tasks.