Cross-Modal Complementarity Learning for Fish Feeding Intensity Recognition via Audio-Visual Fusion

Area of Science:

• Aquaculture technology
• Bioacoustics
• Computer vision

Background:

• Accurate fish feeding intensity evaluation is vital for aquaculture efficiency and fish health.
• Single-modal monitoring (audio or visual) faces challenges in complex underwater environments like turbidity, poor lighting, occlusion, and noise.
• Existing multi-modal fusion methods often use simple strategies, failing to leverage complementary data or dynamically assess reliability.

Purpose of the Study:

• To develop an advanced cross-modal learning framework for robust underwater behavioral analysis.
• To enhance the accuracy of fish feeding intensity evaluation by effectively integrating acoustic and visual data.
• To address limitations of single-modal and conventional fusion techniques in challenging aquatic conditions.

Main Methods:

• Proposed the Adaptive Cross-modal Attention Fusion Network (ACAF-Net), a framework utilizing a two-stage attention fusion mechanism.
• Implemented a cross-modal enhancement stage with Low-rank Bilinear Pooling and learnable fusion weights.
• Developed an adaptive attention fusion stage for dynamic weighting of acoustic and visual features based on reliability and complementarity, incorporating dimension alignment and attention mechanisms.

Main Results:

• ACAF-Net demonstrated superior performance compared to single-modal and conventional fusion approaches.
• Achieved a significant 6.4% improvement in accuracy for fish feeding intensity evaluation.
• Validated the effectiveness of cross-modal complementarity for analyzing underwater behaviors.

Conclusions:

• The proposed ACAF-Net framework effectively exploits cross-modal complementarity for enhanced underwater behavioral analysis.
• The adaptive fusion mechanism improves monitoring reliability in complex aquatic environments.
• This work provides a foundation for developing intelligent aquaculture monitoring systems.