Multi-Modality Adaptive Feature Fusion Graph Convolutional Network for Skeleton-Based Action Recognition.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Graph convolutional networks (GCNs) excel at non-Euclidean data, making them suitable for skeleton-based action recognition.
• Traditional multi-scale temporal convolutions use fixed parameters, which may not be optimal for diverse datasets and network layers.

Purpose of the Study:

• To develop an adaptive framework for skeleton-based action recognition that overcomes limitations of fixed receptive fields and context aggregation.
• To enhance the performance of graph convolutional networks in action recognition tasks.

Main Methods:

• Proposed a multi-modality adaptive feature fusion framework (MMAFF) incorporating adaptive convolution kernels and dilation rates with a self-attention mechanism.
• Introduced a feature fusion mechanism to replace residual connections, improving context aggregation and initial feature fusion.
• Utilized a limb stream to process correlated multi-modal data within the MMAFF.

Main Results:

• The MMAFF framework adaptively selects convolution parameters, optimizing receptive fields across different network layers.
• The feature fusion mechanism effectively addresses context aggregation and initial feature fusion challenges.
• The model achieved competitive results on the NTU-RGB+D 60 and NTU-RGB+D 120 datasets.

Conclusions:

• The proposed MMAFF framework demonstrates superior performance in skeleton-based action recognition by enhancing spatial and temporal receptive fields.
• Adaptive mechanisms and improved feature fusion lead to more robust context aggregation and better recognition accuracy.