iSEARLE: Improving Textual Inversion for Zero-Shot Composed Image Retrieval.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Composed Image Retrieval (CIR) aims to find images similar to a reference image, modified by a relative caption.
• Supervised CIR methods are limited by the need for extensive, manually labeled datasets.
• A new task, Zero-Shot CIR (ZS-CIR), is proposed to address CIR without labeled training data.

Purpose of the Study:

• To introduce and address the Zero-Shot Composed Image Retrieval (ZS-CIR) task.
• To propose an effective method for ZS-CIR that does not rely on labeled training data.
• To present a new benchmark dataset for ZS-CIR research.

Main Methods:

• Developed iSEARLE (improved zero-Shot composEd imAge Retrieval with textuaL invErsion), a novel approach for ZS-CIR.
• Mapped reference image visual information into CLIP's pseudo-word token embedding space.
• Combined visual embeddings with relative captions for retrieval.

Main Results:

• iSEARLE achieved state-of-the-art performance across three CIR datasets: FashionIQ, CIRR, and the new CIRCO dataset.
• The method demonstrated effectiveness in domain conversion and object composition evaluation settings.
• Introduced CIRCO, an open-domain benchmark dataset for CIR with multiple ground truths and semantic categorization.

Conclusions:

• The proposed iSEARLE method effectively performs ZS-CIR, overcoming the need for labeled training data.
• The development of the CIRCO dataset facilitates further research in ZS-CIR.
• The approach shows strong generalization capabilities across various CIR tasks and datasets.