Cross modal recipe retrieval with fine grained modal interaction

Designing systems capable of finding relevant cooking recipes given a user-submitted food image, or vice versa, cross-modal recipe retrieval has gained significant attention in recent years. Numerous advanced techniques have been employed to improve the performance of cross-modal recipe retrieval on general benchmarks. However, leveraging the fine-grained modalities interaction for enhancing multi-modal representation is still limited. Preceding a hierarchical recipe Transformer for encoding individual recipe components, we introduce the cross-component multiscale recipe enriching (CCMRE) module, which enhances the components of the recipe through fully convolutional operations with convolutional kernels of different lengths. Further, we embed a text-contextualized visual enhancing (TCVE) module into an intermediate layer of the image encoder to enrich the visual encoder. Utilizing the similarity between image local features and intermediate recipe representations, TCVE enhances visual representation by deeper model relearning. We conduct a thorough analysis and ablation studies to validate the proposed method, FMI (Fine-grained Modalities Interaction for Cross-Modal Recipe Retrieval). As a result, our method outperforms current SoTA across all metrics on the Recipe1M dataset. Specifically, compared to baseline model, the improvements of + 17.4 R@1 and + 20.5R@1 on the 1 k and 10 k test sets are achieved respectively.