View Adaptive Neural Networks for High Performance Skeleton-Based Human Action Recognition.

Area of Science:

• Computer Vision
• Machine Learning
• Human-Computer Interaction

Background:

• Skeleton-based human action recognition is gaining traction due to the availability of 3D skeleton data.
• A major challenge is handling variations in action representation across different viewpoints.

Purpose of the Study:

• To develop a novel view adaptation scheme to mitigate the impact of viewpoint variations in skeleton-based action recognition.
• To automatically determine optimal virtual observation viewpoints in a data-driven manner.

Main Methods:

• Introduced two view adaptive neural networks: VA-RNN (Recurrent Neural Network with Long Short-Term Memory) and VA-CNN (Convolutional Neural Network).
• Developed a view adaptation module within each network to learn and transform skeletons to suitable viewpoints.
• Implemented a two-stream VA-fusion scheme to combine predictions from both networks.
• Utilized random rotation of skeleton sequences to enhance model robustness and prevent overfitting.

Main Results:

• View adaptive models successfully transform skeletons to consistent virtual viewpoints, minimizing view influence.
• The approach enables networks to focus on action-specific features, leading to superior recognition performance.
• The VA-fusion scheme further enhanced prediction accuracy.
• Experimental evaluations on five benchmarks confirmed the effectiveness and superiority over state-of-the-art methods.

Conclusions:

• The proposed view adaptation scheme effectively addresses viewpoint variations in skeleton-based action recognition.
• The data-driven approach significantly improves recognition accuracy and robustness.
• The method offers a promising direction for advancing human action recognition technologies.