Hollywood 3D: What are the Best 3D Features for Action Recognition?

Area of Science:

• Computer Vision and Pattern Recognition
• Human Action Recognition
• 3D Data Analysis

Background:

• Action recognition "in the wild" is challenging due to information loss when projecting 3D actions to 2D image planes.
• Emerging 3D data from broadcast content and depth sensors offer potential solutions, yet optimal exploitation methods are underexplored.

Purpose of the Study:

• To introduce the Hollywood 3D benchmark, the first dataset for "in the wild" action recognition with integrated 3D data.
• To propose and evaluate novel depth-aware techniques for enhancing action recognition performance.

Main Methods:

• Developed the Hollywood 3D benchmark: 650 stereo video clips (14 action classes) from movies, with stereo calibrations and depth reconstructions.
• Implemented an action recognition pipeline incorporating specialized depth-aware techniques: five interest point detectors and three feature descriptors.
• Conducted extensive tests evaluating various appearance and depth encoding schemes.

Main Results:

• The proposed depth-aware techniques achieved performance levels over three times higher than baseline algorithms relying solely on appearance information.
• Demonstrated the effectiveness of exploiting 3D depth data for significantly improving action recognition accuracy in unconstrained environments.

Conclusions:

• The Hollywood 3D benchmark provides a valuable resource for advancing "in the wild" action recognition research.
• Depth-aware methods represent a significant advancement, substantially outperforming traditional appearance-based approaches.
• The benchmark data, code, and calibrations are publicly released to foster community development.