Convolution-Enhanced Bi-Branch Adaptive Transformer With Cross-Task Interaction for Food Category and Ingredient...

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Visual food analysis is crucial for applications like diet management and personalized recommendations.
• Food images present challenges due to complex, unstructured visual patterns and inherent semantic relationships between ingredients and categories.

Purpose of the Study:

• To develop a multi-task learning framework for accurate food category and ingredient recognition.
• To effectively model long-range semantic relationships and category-ingredient interactions in food images.

Main Methods:

• Proposed Convolution-Enhanced Bi-Branch Adaptive Transformer (CBiAFormer) with Bi-Branch Adaptive Attention (BiA-Attention) for local and global feature extraction.
• Introduced a convolutional patch embedding module to capture fine-grained features.
• Developed Structural Learning and Cross-Task Interaction (SLCI) network utilizing cross-layer attention and cross-task modules for semantic modeling.

Main Results:

• Achieved competitive performance on three mainstream food datasets (ETH Food-101, Vireo Food-172, ISIA Food-200).
• Demonstrated effectiveness through visualization analyses of CBiAFormer and SLCI.

Conclusions:

• The proposed multi-task learning framework effectively addresses challenges in visual food analysis.
• The CBiAFormer and SLCI components significantly improve food category and ingredient recognition accuracy.