Human Action Recognition and Note Recognition: A Deep Learning Approach Using STA-GCN

Area of Science:

• Machine Learning
• Computer Vision
• Music Information Retrieval

Background:

• Human action recognition (HAR) is crucial for various applications.
• Simultaneously recognizing human actions and musical notes in performances presents a significant challenge.
• Existing methods often struggle with the complexity of musical performances.

Purpose of the Study:

• To propose a deep learning-based method for simultaneous human action and musical note recognition.
• To address the challenges of HAR in music performances, specifically on the Morin khuur.
• To develop a model that can accurately interpret both physical movements and musical output.

Main Methods:

• Creation of a new dataset for Morin khuur performances using motion capture and depth sensors.
• Analysis of RGB, depth, and motion data to identify valuable features for recognition.
• Implementation of a Spatial Temporal Attention Graph Convolutional Network (STA-GCN) for gesture recognition.

Main Results:

• The proposed STA-GCN model demonstrated superior performance compared to the traditional ST-GCN.
• The model achieved a high accuracy of 81.4% in recognizing actions and musical notes.
• Feature analysis indicated the importance of hand keypoints and instrument segmentation.

Conclusions:

• The developed deep learning method effectively enables simultaneous HAR and musical note recognition.
• The STA-GCN model shows promise for applications in music performance analysis and HCI.
• This research contributes a novel dataset and a robust model for a complex recognition task.