Robust action recognition via borrowing information across video modalities.

Area of Science:

• Computer Vision
• Machine Learning

Background:

• Advanced imaging devices like depth and binocular cameras offer richer data than traditional RGB cameras for video analysis.
• Limitations such as short effective distances and high costs hinder the practical application of these next-generation cameras.

Purpose of the Study:

• To improve human action recognition accuracy in RGB videos by leveraging data from RGB-Depth (RGB-D) cameras.
• To propose a method that overcomes the practical limitations of RGB-D cameras in real-world applications like surveillance.

Main Methods:

• An offline database was created containing synchronized RGB videos, depth maps, and skeleton data of human actions.
• A novel approach was developed to adapt inter-database variations and enable knowledge transfer across different video modalities.
• RGB action representations were augmented with features borrowed from depth and skeleton data.

Main Results:

• The proposed method demonstrated significant improvements in action recognition accuracy across five benchmark datasets.
• Borrowing complementary information from depth and skeleton data proved effective in enhancing RGB-based action recognition.
• The approach successfully adapted to variations between different data sources.

Conclusions:

• Leveraging offline collected multimodal data (RGB, depth, skeleton) offers a viable solution to enhance action recognition.
• The proposed knowledge borrowing technique effectively overcomes the limitations of using depth cameras directly in practical scenarios.
• This method provides a robust and accurate approach for action recognition, applicable even with conventional RGB data.