Modality Compensation Network: Cross-Modal Adaptation for Action Recognition.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Multimodal video data from RGB-D cameras are increasingly available for human action recognition.
• Effectively leveraging complementary information between different modalities is a key challenge.

Purpose of the Study:

• To propose a Modality Compensation Network (MCN) for human action recognition.
• To explore relationships between modalities and boost representations using complementary information.

Main Methods:

• Utilized RGB/optical flow videos as source modalities and skeletons as auxiliary modality.
• Developed a network combining Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) networks.
• Implemented a modality adaptation block for adaptive representation learning, compensating for skeleton data loss.

Main Results:

• The MCN model demonstrated improved recognition performance using only source data during testing.
• Explored multiple adaptation schemes to align source and auxiliary modal distributions.
• Achieved state-of-the-art results on four widely-used action recognition benchmarks.

Conclusions:

• The proposed MCN effectively leverages auxiliary skeleton data to enhance human action recognition from source modalities.
• The network can compensate for missing skeleton data, enabling robust performance.
• MCN offers a promising approach for multimodal human action recognition tasks.