Revisit Weakly Supervised Hashing With Deep Multi-Modal Foundation Models.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Vision-Language Pretraining (VLP) models excel in multimodal tasks but are underexplored for large-scale image retrieval.
• Real-world image retrieval systems utilize web-scraped images with user-annotated tags, offering potential for weak supervision.
• Existing methods have limited exploration of VLP models for enhancing compact image representations in retrieval.

Purpose of the Study:

• To harness the image-and-text alignment capabilities of VLP foundation models for enhanced compact image representation.
• To develop a novel weakly supervised hashing framework for large-scale image retrieval.
• To improve the performance of image retrieval systems by leveraging weak supervision from user-annotated tags.

Main Methods:

• Propose a weakly supervised hashing framework that iteratively learns a deep hashing network and enhances weak supervision.
• Extract image and tag representations from VLP foundation models.
• Employ a policy gradient process to optimize retrieval performance (mAP) and a probabilistic decision process to refine supervision.

Main Results:

• The proposed framework effectively enhances compact image representation for large-scale image retrieval.
• Experiments on public datasets demonstrate the superiority of the developed method compared to existing approaches.
• The alternative optimization of the hashing network and weak supervision leads to significant improvements in retrieval metrics.

Conclusions:

• The developed weakly supervised hashing framework successfully leverages VLP models for improved image retrieval.
• This approach offers a promising direction for utilizing web-scraped data and VLP models in large-scale image retrieval systems.
• The method provides a robust and effective solution for learning compact image representations through weak supervision.