Benchmarking and Improving Foundation Model Dietary Estimates from Meal Images

Area of Science:

• Artificial Intelligence
• Computer Vision
• Nutritional Science

Background:

• Accurate dietary content quantification (calories, protein, carbs, fats) from meal images is crucial for diabetes management.
• Large Multimodal Models (LMMs) demonstrate advanced capabilities in complex vision-language tasks due to extensive, diverse training data.

Purpose of the Study:

• To benchmark seven Large Multimodal Models (LMMs) against a specialized RGB-D fusion model for nutritional estimation.
• To evaluate the generalization capacity of LMMs on diverse datasets.
• To develop a method for integrating physical size information into phone-collected meal images for improved dietary assessment.

Main Methods:

• Benchmarking seven LMMs (GPT, Gemini, Llama) and an RGB-D fusion model on the Nutrition5k and DonateAndLearn datasets.
• Analyzing LMM performance and generalization capacity.
• Proposing and applying a method to integrate scaled phone images into the RGB-D fusion model to predict food weight.

Main Results:

• Full-weight LMMs significantly outperformed the RGB-D fusion model on the DonateAndLearn dataset, indicating superior generalization.
• Integrating predicted food weight with the Gemini 2.5 Flash model reduced carbohydrate prediction MAPE from 56.6% to 39.5%.
• Providing ground-truth food weight to Gemini 2.5 Flash and GPT-4.1 dramatically improved MAPE to 20.2% and 26.8%, respectively.

Conclusions:

• LMMs exhibit strong potential for accurate dietary content estimation from meal images.
• Incorporating physical size information is critical for enhancing the accuracy of dietary assessment tools.
• These findings support the development of advanced AI-driven tools for diabetes management and nutritional tracking.