CoVR-2: Automatic Data Construction for Composed Video Retrieval.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Composed Image Retrieval (CoIR) is a popular task involving text and image queries for image search.
• Existing CoIR methods rely on expensive, manually annotated datasets (image-text-image triplets).
• Manual data curation limits the scalability of CoIR approaches.

Purpose of the Study:

• To develop a scalable, automatic methodology for creating datasets for Composed Video Retrieval (CoVR) and CoIR.
• To expand the scope of retrieval tasks to include video data.
• To construct large-scale datasets for training and evaluating retrieval models.

Main Methods:

• Leveraged video-caption pairs to automatically generate triplets for CoIR and CoVR.
• Employed large language models to generate modification text for triplets.
• Utilized the WebVid2M and Conceptual Captions datasets to create large training datasets (WebVid-CoVR and CoIR triplets).
• Adapted BLIP-2 pretraining for composed retrieval and incorporated a caption retrieval loss.

Main Results:

• Created the WebVid-CoVR dataset with 1.6 million triplets and 3.3 million CoIR training triplets.
• Introduced a new benchmark for CoVR with baseline results.
• Demonstrated effective transfer learning from CoVR models trained on the new datasets to CoIR tasks.
• Achieved improved state-of-the-art performance in zero-shot retrieval on CIRR, FashionIQ, and CIRCO benchmarks.

Conclusions:

• The proposed automatic dataset creation methodology is scalable and effective for CoIR and CoVR.
• The new datasets and benchmark facilitate research in composed retrieval.
• The approach significantly enhances retrieval performance, particularly in zero-shot settings.