Reasoning-Driven Food Energy Estimation via Multimodal Large Language Models.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Nutritional Science

Background:

• Accurate image-based food energy estimation is crucial for dietary tracking applications on smartphones and AR devices.
• Current deep learning methods face challenges in food item recognition due to extensive data annotation requirements.
• Multimodal Large Language Models (MLLMs) offer potential but struggle with accurate food energy estimation due to difficulties in recognizing food size.

Purpose of the Study:

• To enhance the accuracy of image-based food energy estimation using Multimodal Large Language Models (MLLMs).
• To address the limitation of food size recognition in MLLMs for improved energy content assessment.
• To explore fine-tuning and volume-aware reasoning strategies for more precise dietary intake monitoring.

Main Methods:

• Proposed two novel approaches: fine-tuning MLLMs and employing volume-aware reasoning with fine-grained estimation prompting.
• Investigated the adaptation of Low-Rank Adaptation (LoRA) to further boost food energy estimation performance.
• Utilized the Nutrition5k dataset for experimental validation and performance evaluation.

Main Results:

• Both proposed fine-tuning and volume-aware reasoning approaches significantly improved food energy estimation accuracy.
• Experimental results on the Nutrition5k dataset confirmed the effectiveness of the developed methods.
• Adapting LoRA demonstrated a positive impact on enhancing overall food energy estimation capabilities.

Conclusions:

• MLLMs show significant promise for accurate image-based dietary assessment.
• Integrating volume-awareness is critical for developing robust food energy estimation models.
• The study highlights effective strategies for leveraging MLLMs in nutritional applications.