Contrastive self-supervised representation learning without negative samples for multimodal human action recognition.

Area of Science:

• Computer Vision
• Machine Learning
• Human-Computer Interaction

Background:

• Action recognition is crucial for human-computer interaction.
• Current methods, especially Convolutional Neural Networks (ConvNets), are limited by insufficient large-scale labeled data.
• Multimodal feature representation offers potential improvements due to complementary data sources.

Purpose of the Study:

• To propose a novel multimodal feature representation and contrastive self-supervised learning framework.
• To enhance action recognition performance and model generalization capabilities.
• To address the limitations of data scarcity in existing methods.

Main Methods:

• A novel framework utilizing weight sharing between two branches for multimodal feature learning.
• Contrastive self-supervised learning approach that does not require negative samples.
• Integration of skeleton sequences and inertial measurement unit (IMU) signals for robust feature extraction.

Main Results:

• The proposed framework effectively learns useful representations from unlabeled multimodal data.
• Demonstrated superior performance compared to unimodal and multimodal baselines.
• Achieved state-of-the-art results on UTD-MHAD and MMAct benchmarks across action retrieval, semi-supervised, and zero-shot learning.

Conclusions:

• The developed framework offers an effective solution for action recognition with limited labeled data.
• Multimodal self-supervised learning is a promising direction for improving model generalization.
• The approach successfully leverages complementary information from skeleton and IMU data.