Cross-modal recipe retrieval via multi-granularity alignment.

Area of Science:

• Food Computing
• Artificial Intelligence
• Computer Vision

Background:

• Cross-modal recipe retrieval is crucial for food safety and nutrition management.
• Existing methods struggle with ingredient alignment between recipe images and text, leading to retrieval inaccuracies.
• Distinguishing recipes with similar ingredients remains a significant challenge.

Purpose of the Study:

• To propose a novel Cross-modal Multi-Granularity Alignment (CMGA) method for enhanced recipe retrieval.
• To address the limitations of previous methods in aligning key food ingredients across modalities.
• To improve the accuracy and granularity of cross-modal recipe searching.

Main Methods:

• Developed a Key Information Enhancement (KIE) module using cross-attention and Large Language Models (LLMs) to focus on explicit food ingredients in text.
• Introduced an Image Detail Enhancement (IDE) module with sparse attention and dynamic feature fusion to capture visual food components and cooking details.
• Implemented a bidirectional contrastive loss function to emphasize learning from challenging, similar recipe examples.

Main Results:

• The proposed CMGA method demonstrated superior performance compared to baseline models on the Recipe1M dataset.
• CMGA effectively improved the alignment of main food ingredients between textual descriptions and visual representations.
• The method showed enhanced capability in distinguishing between recipes with subtle ingredient differences.

Conclusions:

• The CMGA method offers a significant advancement in cross-modal recipe retrieval.
• Focusing on multi-granularity alignment, particularly ingredient-level details, is key to overcoming previous limitations.
• This research contributes to more accurate and reliable food information retrieval systems.