A Unified Multimodal De- and Re-Coupling Framework for RGB-D Motion Recognition

Area of Science:

• Computer Vision
• Machine Learning
• Multimodal Learning

Background:

• Video classification models face challenges due to limited data and complex parameters.
• Existing RGB-D based motion recognition methods have limitations in data augmentation, spatiotemporal modeling, and cross-modal fusion.

Purpose of the Study:

• To improve RGB-D based motion recognition by addressing data limitations and algorithmic challenges.
• To introduce novel data augmentation and fusion techniques for enhanced video representation learning.

Main Methods:

• Introduced ShuffleMix, a novel video data augmentation method for temporal regularization.
• Proposed the Unified Multimodal De-coupling and Re-coupling (UMDR) framework for video representation learning.
• Developed a Complement Feature Catcher (CFCer) for improved late fusion of multimodal information.

Main Results:

• The proposed methods significantly enhance spatiotemporal representation learning.
• Achieved superior performance compared to state-of-the-art methods on four public motion datasets.
• Demonstrated an improvement of 4.5% on the Chalearn IsoGD dataset using the UMDR framework.

Conclusions:

• The integrated approach of ShuffleMix and UMDR framework offers a robust solution for RGB-D based motion recognition.
• The novel techniques effectively address limitations in data augmentation, optimization, and cross-modal fusion.
• The study advances the field of motion recognition with significant performance gains.