IGSMNet: Ingredient-Guided Semantic Modeling Network for Food Nutrition Estimation

Area of Science:

• Computer Vision
• Artificial Intelligence
• Nutritional Science

Background:

• Traditional food nutrition estimation is labor-intensive and lacks scalability.
• Existing computer vision methods, including RGB and RGB-D approaches, struggle with visually similar foods and complex spatial-semantic relationships.
• Accurate nutritional assessment is vital for dietary analysis and public health initiatives.

Purpose of the Study:

• To develop an advanced food nutrition estimation method that overcomes limitations of current techniques.
• To improve the accuracy of estimating nutritional content in complex food scenes.
• To leverage ingredient information and spatial-semantic understanding for enhanced prediction.

Main Methods:

• Proposed an Ingredient-Guided Semantic Modeling Network (IGSMNet) for food nutrition estimation.
• Integrated an ingredient-guided module using a pre-trained language model and cross-modal attention to align ingredient and visual features.
• Implemented an internal semantic modeling component with dynamic positional encoding and localized attention for enhanced structural and relational understanding.

Main Results:

• Achieved state-of-the-art performance on the Nutrition5k dataset.
• Reported low Percentage Mean Absolute Error (PMAE) values: 12.2% for Calories, 9.4% for Mass, 19.1% for Fat, 18.3% for Carbohydrates, and 16.0% for Protein.
• Demonstrated consistent outperformance compared to existing baseline methods.

Conclusions:

• The IGSMNet effectively addresses challenges in complex food scenes for nutrition estimation.
• The proposed method shows significant improvements in accuracy for estimating various nutritional components.
• This work validates the effectiveness of integrating ingredient-specific and semantic spatial information for precise food nutrition analysis.