Benchmarking and Improving Foundation Model Dietary Estimates from Meal Images

Area of Science:

• Artificial Intelligence
• Computer Vision
• Nutritional Science

Background:

• Accurate dietary content quantification (calories, macronutrients) from meal images is crucial for diabetes management.
• Large Multimodal Models (LMMs) demonstrate advanced capabilities in complex vision-language tasks due to extensive training data.

Purpose of the Study:

• To benchmark seven Large Multimodal Models (LMMs) against a traditional RGB-D fusion model for nutritional estimation from meal images.
• To evaluate the generalization capacity of LMMs on diverse datasets.
• To develop a method for integrating physical size information into image-based dietary assessments.

Main Methods:

• Benchmarking seven LMMs (GPT, Gemini, Llama) and an RGB-D fusion model on the Nutrition5k and DonateAndLearn datasets.
• Analyzing LMM performance and generalization capabilities.
• Proposing and applying a method to integrate scaled phone images for food weight prediction.

Main Results:

• Full-weight LMMs significantly outperformed the RGB-D fusion model on the DonateAndLearn dataset, indicating superior generalization.
• Integrating predicted food weight with the Gemini 2.5 Flash model reduced carbohydrate prediction MAPE from 56.6% to 39.5%.
• Providing ground-truth food weight further improved MAPE to 20.2% (Gemini 2.5 Flash) and 26.8% (GPT-4.1).

Conclusions:

• LMMs offer superior performance and generalization for image-based nutritional estimation compared to traditional models.
• Incorporating physical size information is critical for enhancing the accuracy of dietary assessment tools.
• Future dietary assessment tools should leverage LMMs and integrate physical dimensions for improved accuracy.