Hybrid Decoding with Co-Occurrence Awareness for Fine-Grained Food Image Segmentation

Area of Science:

• Computer Vision
• Artificial Intelligence
• Biomedical Informatics

Background:

• Fine-grained food image segmentation is critical for dietary assessment and nutritional analysis.
• Challenges include ambiguous boundaries, similar food appearances, and complex meal layouts.
• Current CNN, Transformer, and Mamba models struggle with local details and long-range context.

Purpose of the Study:

• To develop a novel decoding framework, Hybrid Decoder for Food Image Segmentation (HDF), to enhance food image segmentation.
• To address limitations of existing methods in capturing both fine-grained details and contextual dependencies.
• To improve the accuracy of segmenting diverse food ingredients in complex meal images.

Main Methods:

• Proposed HDF framework utilizing the MambaVision backbone.
• Employed a convolution-based Feature Pyramid Network (FPN) for multi-stage feature extraction.
• Integrated a Cross-Layer Mamba module for efficient multi-scale feature fusion.
• Incorporated an Attention Refinement module for global semantic context.
• Introduced a Food Co-occurrence Module to learn category patterns and enhance semantics.

Main Results:

• Achieved 52.25% mIoU on the FoodSeg103 benchmark.
• Achieved 76.16% mIoU on the UEC-FoodPIX Complete benchmark.
• Outperformed existing state-of-the-art methods on standard food segmentation datasets.

Conclusions:

• The HDF framework effectively addresses challenges in fine-grained food image segmentation.
• The hybrid design and co-occurrence module significantly improve segmentation accuracy.
• HDF provides a robust foundation for applications in dietary logging, nutritional estimation, and food safety.