Weakly Supervised Micro- and Macro-Expression Spotting Based on Multi-Level Consistency | JoVE Visualize

Area of Science:

Computer Science
Artificial Intelligence
Machine Learning

Background:

Micro- and macro-expression spotting in videos is challenging due to extensive data collection and frame-level annotation requirements.
Existing weakly supervised expression spotting (WES) methods, often based on multiple instance learning (MIL), face significant inter-modality, inter-sample, and inter-task gaps.
The inter-sample gap, particularly concerning sample distribution and duration, hinders performance in current WES approaches.

Purpose of the Study:

To propose a novel and simplified WES framework, MC-WES, designed to achieve fine-grained frame-level spotting using only video-level labels.
To address the limitations of existing WES methods by mitigating various inter-gap issues and integrating prior knowledge.
To develop a system that alleviates the burden of frame-wise annotation while maintaining high spotting accuracy.

Main Methods:

MC-WES employs multi-consistency collaborative mechanisms, including modal-level saliency, video-level distribution, label-level duration, and segment-level feature consistency strategies.
Modal-level saliency consistency captures correlations between raw images and optical flow.
Video-level distribution consistency leverages temporal sparsity differences; label-level duration consistency exploits facial muscle duration variations; segment-level feature consistency ensures similarity for features under identical labels.

Main Results:

MC-WES demonstrates effective fine frame-level spotting capabilities using only video-level labels.
The proposed multi-consistency strategies successfully alleviate identified gaps in existing WES methods.
Experimental results on CAS(ME)$^{2}$, CAS(ME)$^{3}$, and SAMM-LV datasets show MC-WES performance comparable to state-of-the-art fully supervised methods.

Conclusions:

MC-WES offers a simplified yet powerful approach to weakly supervised expression spotting, reducing annotation complexity.
The multi-consistency framework effectively merges prior knowledge and addresses inherent gaps in MIL-based WES.
MC-WES achieves competitive performance, suggesting its viability as an alternative to fully supervised methods for expression spotting tasks.