Multimodal Action Quality Assessment.

Area of Science:

• Computer Vision
• Machine Learning
• Signal Processing

Background:

• Action Quality Assessment (AQA) traditionally relies solely on visual data.
• Audio information, particularly in music-accompanied sports, is underutilized in AQA.
• Existing methods may not optimally leverage multimodal data for improved accuracy.

Purpose of the Study:

• To propose a novel multimodal fusion network for enhanced Action Quality Assessment (AQA).
• To investigate the complementary role of audio information alongside visual data (RGB, optical flow) in AQA.
• To develop an adaptive fusion mechanism that accounts for action diversity.

Main Methods:

• Proposed the Progressive Adaptive Multimodal Fusion Network (PAMFN) with modality-specific and mixed-modality branches.
• Introduced a Modality-specific Feature Decoder for selective information transfer.
• Developed an Adaptive Fusion Module with FusionNets and a PolicyNet for dynamic fusion strategies.
• Implemented a Cross-modal Feature Decoder to integrate fused features.

Main Results:

• The proposed PAMFN effectively models modality-specific and mixed-modality information.
• Experimental results demonstrate the efficacy of the adaptive fusion strategy.
• The method achieved state-of-the-art performance on two public AQA datasets.

Conclusions:

• Integrating audio information significantly enhances action quality assessment accuracy.
• The proposed adaptive multimodal fusion approach is superior to invariant fusion policies.
• PAMFN offers a robust and effective solution for state-of-the-art AQA.