AquaticCLIP: A Vision-Language Foundation Model and Dataset for Underwater Scene Analysis

Area of Science:

• Marine biology
• Computer vision
• Artificial intelligence

Background:

• Aquatic biodiversity is vital for climate change mitigation.
• Understanding aquatic scenes aids marine scientists' decision-making.
• Existing vision-language models (VLMs) lack specialized aquatic domain adaptation.

Purpose of the Study:

• Introduce AquaticCLIP, a novel contrastive language-image pretraining (CLIP) model for aquatic scene understanding.
• Develop an underwater domain-specific learning framework.
• Improve zero-shot performance in underwater computer vision tasks.

Main Methods:

• Constructed a 2-million image-text paired dataset from diverse aquatic resources.
• Proposed a prompt-guided vision encoder (PGVE) for feature aggregation.
• Implemented a vision-guided mechanism to enhance the language encoder.
• Optimized the model using contrastive pretraining loss.

Main Results:

• AquaticCLIP achieves significant performance improvements in zero-shot underwater computer vision tasks.
• The model demonstrates superior accuracy and robustness compared to existing methods.
• Sets a new benchmark for vision-language applications in aquatic environments.

Conclusions:

• AquaticCLIP effectively aligns visual and textual modalities in aquatic environments.
• The model enhances capabilities in segmentation, classification, detection, and object counting.
• Publicly available code and dataset facilitate further research in underwater vision-language understanding.