Sample Fusion Network: An End-to-End Data Augmentation Network for Skeleton-Based Human Action Recognition.

Area of Science:

• Computer Vision
• Machine Learning
• Deep Learning

Background:

• Data augmentation is crucial for deep neural networks in skeleton-based human action recognition (HAR).
• Existing handcrafted augmentation methods lack learnable parameters, limiting their effectiveness during training and testing.
• There is a need for adaptable and trainable data augmentation techniques for HAR.

Purpose of the Study:

• To propose a novel Sample Fusion Network (SFN) for data augmentation in HAR.
• To develop an end-to-end trainable framework by integrating SFN with HAR networks.
• To enhance the performance and generalization ability of skeleton-based HAR models.

Main Methods:

• A Sample Fusion Network (SFN) utilizing a Long Short-Term Memory (LSTM) autoencoder (AE) was developed.
• The SFN generates new samples by learning data distributions, unlike handcrafted methods.
• An adaptive weighting strategy was employed to optimize sample complementarity and improve HAR performance.

Main Results:

• The proposed SFN, when integrated with a baseline HAR model, increased classification accuracy from 79.53% to 90.75% on the NTU RGB+D dataset.
• Experimental results demonstrated superior performance compared to state-of-the-art data augmentation methods.
• The SFN framework showed significant improvements in HAR tasks across various datasets.

Conclusions:

• The Sample Fusion Network (SFN) offers a powerful and generalizable approach to data augmentation for skeleton-based HAR.
• End-to-end training of SFN with HAR networks leads to substantial performance gains.
• The proposed method effectively addresses limitations of traditional augmentation techniques and advances the field of HAR.