A Unified Deep Framework for Joint 3D Pose Estimation and Action Recognition from a Single RGB Camera

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Accurate 3D human pose estimation and action recognition are crucial for intelligent systems.
• Existing methods often rely on expensive RGB-depth sensors or have high computational demands.
• Leveraging widely available RGB cameras for these tasks remains a significant challenge.

Purpose of the Study:

• To develop a cost-effective and computationally efficient deep learning framework for joint 3D human pose estimation and action recognition using monocular RGB sensors.
• To achieve performance comparable to RGB-depth sensor-based methods using only standard RGB cameras.
• To explore the potential of leveraging ubiquitous RGB cameras for advanced human behavior analysis.

Main Methods:

• A two-stage deep learning approach is proposed.
• Stage 1: Real-time 2D pose detection identifies human keypoints, followed by a two-stream neural network mapping 2D keypoints to 3D poses.
• Stage 2: Efficient Neural Architecture Search (ENAS) optimizes a network for spatio-temporal modeling of 3D poses and action recognition using an image-based intermediate representation.

Main Results:

• The framework successfully performs joint 3D human pose estimation and action recognition from RGB data.
• Experiments on Human3.6M, MSR Action3D, and SBU Kinect Interaction datasets demonstrate the method's effectiveness.
• The approach achieves performance on par with RGB-depth sensors while requiring a low computational budget for training and inference.

Conclusions:

• The proposed deep learning framework enables high-performance 3D human pose estimation and action recognition using affordable monocular RGB cameras.
• This research significantly reduces the hardware cost and computational requirements for advanced human behavior analysis.
• The findings open avenues for deploying intelligent recognition systems in diverse real-world scenarios leveraging existing RGB camera infrastructure.