On the Limitations of Visual-Semantic Embedding Networks for Image-to-Text Information Retrieval

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Visual-semantic embedding (VSE) networks generate joint image-text representations for cross-modal retrieval tasks.
• State-of-the-art VSE networks include VSE++, SCAN, VSRN, and UNITER.
• Evaluating these networks is crucial for advancing image-text retrieval capabilities.

Purpose of the Study:

• To evaluate the performance of leading VSE networks on image-to-text retrieval.
• To identify and analyze the strengths and limitations of these VSE networks.
• To provide insights for future research in cross-modal information retrieval.

Main Methods:

• Performance evaluation of VSE networks (VSE++, SCAN, VSRN, UNITER) on the Flickr30K dataset for image-to-text retrieval.
• Analysis of top-performing models (VSRN, UNITER) using challenging image-text pairs from worst-performing classes.
• Qualitative analysis of limitations based on image scenes, objects, semantics, and neural network functions.

Main Results:

• UNITER achieved the highest average Recall@5 (61.5%) in image-to-text retrieval.
• VSRN, SCAN, and VSE++ achieved 50.3%, 47.1%, and 29.4% Recall@5, respectively.
• Limitations were identified in VSRN and UNITER concerning complex scenes, object details, and semantic nuances.

Conclusions:

• UNITER demonstrates superior performance in image-to-text retrieval compared to traditional VSE networks.
• Understanding model limitations is key to developing more robust VSE networks for cross-modal tasks.
• This research guides future VSE network development for enhanced cross-modal information retrieval.