DietAI24 as a framework for comprehensive nutrition estimation using multimodal large language models

Area of Science:

• Nutritional Science
• Artificial Intelligence
• Computer Vision

Background:

• Accurate dietary assessment is crucial for health research.
• Current smartphone-based food image recognition methods have limitations in analyzing real-world food images and only basic macronutrients.
• Existing computer vision approaches lack the comprehensive nutritional analysis required for in-depth research.

Purpose of the Study:

• To develop an automated nutrition estimation framework from food images.
• To improve the accuracy and comprehensiveness of dietary assessment using AI.
• To overcome the limitations of existing computer vision methods in nutritional research.

Main Methods:

• Developed DietAI24, a framework combining multimodal large language models (MLLMs) with Retrieval-Augmented Generation (RAG).
• Utilized the Food and Nutrient Database for Dietary Studies (FNDDS) as an authoritative nutrition database to ground MLLM visual recognition.
• Enabled accurate nutrient estimation without extensive data collection or model training by leveraging RAG.

Main Results:

• DietAI24 significantly outperforms existing methods and commercial platforms on the ASA24 and Nutrition5k datasets.
• Achieved a 63% reduction in mean absolute error (MAE) for food weight and key nutrient estimation on real-world mixed dishes (p < 0.05).
• Estimates 65 distinct nutrients and food components, surpassing the basic macronutrient analysis of current solutions.

Conclusions:

• Integrating MLLMs with RAG and standardized nutrition databases substantially enhances dietary assessment accuracy and enables comprehensive nutrient analysis.
• DietAI24 offers a scalable solution for nutrition research and clinical applications.
• This framework has the potential to transform large-scale epidemiological studies and personalized dietary interventions through more accurate and less burdensome dietary data collection.