Entity-Graph Enhanced Cross-Modal Pretraining for Instance-Level Product Retrieval.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Instance-level product retrieval in realistic, weakly-supervised settings is challenging.
• Accurately identifying product targets and handling irrelevant content in multi-modal data are key difficulties.
• Existing cross-modal retrieval methods struggle with fine-grained product categories.

Purpose of the Study:

• To develop a more effective cross-modal pretraining model for instance-level product retrieval.
• To address the challenges of identifying specific products and mitigating irrelevant information in visual-linguistic data.
• To enable evaluations on practical tasks like price comparison and personalized recommendations.

Main Methods:

• Introduced the Product1M datasets and two instance-level retrieval tasks.
• Devised a novel Entity-Graph Enhanced Cross-Modal Pretraining (EGE-CMP) model.
• Utilized an entity graph to incorporate key concept information adaptively into multi-modal networks via a self-supervised hybrid-stream transformer.
• Injected entity knowledge in node-based and subgraph-based ways to reduce object confusion.

Main Results:

• The EGE-CMP model effectively guides networks to focus on entities with real semantics.
• Experimental results demonstrate the efficacy and generalizability of the EGE-CMP model.
• EGE-CMP outperforms several state-of-the-art cross-modal baselines, including CLIP, UNITER, and CAPTURE.

Conclusions:

• The proposed EGE-CMP model significantly improves weakly-supervised, multi-modal instance-level product retrieval.
• The entity graph integration enhances the model's ability to discern relevant product information.
• This research provides a robust solution for fine-grained product retrieval in practical applications.