Pose-Appearance Relational Modeling for Video Action Recognition.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Video action recognition methods are typically appearance-based or pose-based.
• Appearance-based methods struggle with temporal dynamics of large motions.
• Pose-based methods often neglect crucial visual context like scenes and objects.

Purpose of the Study:

• To propose a novel Pose-Appearance Relational Network (PARNet) for improved video action recognition.
• To enhance robustness in unconstrained real-world videos by integrating pose and appearance information.
• To leverage the complementary strengths of human pose and visual context for better action understanding.

Main Methods:

• Developed a three-stream network: pose stream (Temporal Multi-Pose RNN), appearance stream (Spatial Appearance CNN), and relation stream (Pose-Aware RNN).
• The relation stream models action-sensitive visual context by connecting pose and appearance streams.
• Jointly optimized the three modules for comprehensive action recognition.

Main Results:

• PARNet achieved superior performance over state-of-the-art methods on both pose-complete and pose-incomplete datasets (KTH, Penn-Action, UCF11, UCF101, HMDB51, JHMDB).
• Demonstrated robustness against complex environments and noisy skeletons.
• An enhanced PARNet with an RGB-based I3D stream further improved performance on UCF101 and HMDB51.

Conclusions:

• The proposed PARNet framework effectively integrates human pose and appearance information for robust video action recognition.
• PARNet shows significant potential for real-world applications requiring reliable action understanding.
• The approach offers a promising direction for future research in video analysis by combining diverse information modalities.