Sound Can Help Us See More Clearly.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Existing video action classification networks primarily rely on visual frames.
• This approach struggles when action-relevant objects are not prominently featured in frames.
• There is a need for multimodal approaches to improve classification robustness.

Purpose of the Study:

• To develop a novel neural network architecture for video action classification that incorporates audio information.
• To enhance the accuracy and robustness of action recognition by utilizing both visual and auditory data.
• To demonstrate the effectiveness of a sound-based approach in complementing frame-based video analysis.

Main Methods:

• Converted original sound waves into sound texture for network input.
• Designed a two-stream network integrating a sound texture-based network with a deep neural network processing video frames.
• Proposed the A-IN (Audio-INtegrated) network architecture.
• Evaluated the A-IN model on the Kinetics dataset, comparing it against an image-only network.

Main Results:

• The proposed A-IN model achieved a 7.6% increase in recognition accuracy compared to image-only networks.
• The integration of sound data features demonstrably improved video action classification performance.
• Multimodal fusion proved effective in overcoming limitations of purely visual methods.

Conclusions:

• Sound data can be effectively utilized to solve motion recognition tasks in videos.
• The A-IN two-stream network demonstrates the benefit of fusing audio and visual information for improved video action classification.
• Leveraging rich multimodal information within videos enhances classification efficacy.