SFAN: Selective Filter and Alignment Network for Cross-Modal Retrieval.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Cross-modal retrieval faces challenges in effectively bridging visual and textual data.
• Fine-grained matching improves performance but struggles with filtering irrelevant multimodal features.
• Minimizing misalignment interference is crucial for accurate cross-modal retrieval.

Purpose of the Study:

• To propose a novel approach, the Selective Filter and Alignment Network (SFAN), for enhanced cross-modal retrieval.
• To address the challenge of filtering irrelevant features within and between modalities.
• To improve the alignment of salient cross-modal features and reduce misalignment interference.

Main Methods:

• Developed modality-specific selective filter modules (SFMs) to implicitly filter redundant information within each modality.
• Introduced a state-space models (SSMs)-based selective alignment module (SAM) for capturing key correspondences.
• Utilized a fusion operation to combine SFM and SAM embeddings for final similarity computation.

Main Results:

• The proposed SFAN effectively learns robust patterns for cross-modal retrieval.
• Experiments on Flickr30k, MS-COCO, and MSR-VTT datasets demonstrate significant performance improvements.
• SFAN outperforms existing state-of-the-art cross-modal retrieval methods.

Conclusions:

• SFAN offers an effective solution for filtering irrelevant features and improving cross-modal alignment.
• The network architecture enhances the robustness and accuracy of cross-modal retrieval.
• This approach represents a significant advancement in the field of cross-modal retrieval.