NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding

Area of Science:

• Computer Vision
• Machine Learning
• Human-Computer Interaction

Background:

• Depth-based and RGB+D methods show promise for human activity analysis.
• Existing datasets lack scale, diversity, and realistic conditions for robust action recognition.

Purpose of the Study:

• Introduce a large-scale RGB+D dataset for human action recognition.
• Address limitations of current benchmarks in terms of data size, class variety, and environmental conditions.

Main Methods:

• Collected a dataset from 106 subjects, comprising over 114,000 videos and 8 million frames.
• Included 120 distinct action classes covering daily, mutual, and health-related activities.
• Evaluated existing 3D activity analysis methods and proposed an Action-Part Semantic Relevance-aware (APSR) framework for one-shot recognition.

Main Results:

• Demonstrated the advantage of deep learning methods for 3D-based human action recognition on the new dataset.
• Achieved promising results for novel action class recognition using the proposed APSR framework.
• The dataset facilitates the development of data-hungry learning techniques for human activity understanding.

Conclusions:

• The large-scale RGB+D dataset is a valuable resource for advancing human activity recognition research.
• The proposed APSR framework shows potential for efficient learning of new action classes.
• This work will spur further innovation in depth-based and RGB+D-based human activity analysis.