Related Experiment Videos
Incomplete Multimodal Probability Flow Recovery for Emotion Recognition
Abstract:
Multimodal emotion recognition (MER) leverages heterogeneous cues to overcome unimodal limitations, yet real-world applications often suffer from modality absence that degrades multimodal fusion effectiveness. While existing recovery-based methods address missing modalities, they face challenges including uncontrollable restoration processes, low-fidelity outputs, and inter-modal inconsistencies. In this work, we propose an Incomplete Multimodal Probability Flow Recovery (IM-PFR) framework to boost modality-missed emotion recognition. By unifying previous approaches into an autoencoder-like paradigm, we derive a Reversible Probability Flow Transformation (RPFT) mode through non-stochastic ordinary differential equations (ODEs), which combines the advantages of controllable sampling and continuous-time modeling. This creates a controllable Available $\leftrightarrow$ Prior$\leftrightarrow$Missing paradigm supporting multiple recovery scenarios (one-to-many, many-to-one, many-to-many), enhanced by two key components: i) a time-dependent aligner coordinating multimodal generation processes, and ii) a cross-modal high-order ODE solver reducing error accumulation. Extending from our prior works, this work enables tractable prior learning while ensuring inherent reversibility without explicit constraints. Extensive experiments on various MER datasets demonstrate state-of-the-art performance, with both quantitative metrics and qualitative analyses confirming superior recovery fidelity across diverse missing-modality conditions.
Related Concept Videos
Labeling Emotion
Uniform Depth Channel Flow: Problem Solving
Physiology of Emotion
Autonomic Nervous System
The autonomic nervous system (ANS) plays a critical role in emotional responses by regulating involuntary physiological functions. It consists of two main components: the sympathetic and parasympathetic systems. The sympathetic system...
Cognitive Theories: Schachter-Singer Theory of Emotion
Physiological Arousal and Cognitive Labeling
According to this theory, when an individual experiences physiological...
Uniform Depth Channel Flow
Cognitive Theories: Lazarus Mediational Theory of Emotion
Cognitive Appraisal and Emotional Response
Lazarus proposed that emotions are not solely...