Disambiguity and Alignment: An Effective Multi-Modal Alignment Method for Cross-Modal Recipe Retrieval.

Area of Science:

• Food computing
• Computer vision
• Information retrieval

Background:

• Cross-modal recipe retrieval is vital in food computing.
• Existing methods lack intra-modal alignment, hindering semantic understanding.
• Food image ambiguity is a critical, overlooked challenge.

Purpose of the Study:

• To propose a novel Multi-Modal Alignment Method for Cross-Modal Recipe Retrieval (MMACMR).
• To enhance semantic alignment by considering both inter-modal and intra-modal alignment.
• To address the issue of food image ambiguity in retrieval models.

Main Methods:

• MMACMR measures ambiguous food image similarity guided by corresponding recipes.
• A cross-attention module is integrated between ingredients and instructions for enhanced recipe representation.
• The method jointly optimizes inter-modal and intra-modal alignment.

Main Results:

• Experiments conducted on the Recipe1M dataset.
• The proposed MMACMR method significantly outperforms several state-of-the-art methods.
• Demonstrated improved performance in commonly used evaluation criteria for cross-modal retrieval.

Conclusions:

• MMACMR effectively enhances semantic alignment for cross-modal recipe retrieval.
• The method successfully addresses food image ambiguity, improving model convergence.
• This work offers a significant advancement in food computing and recipe retrieval systems.