A Confidence-Aware Hybrid Vision-Language Framework for Food Recognition and Nutritional Monitoring

Area of Science:

• Computer Vision
• Artificial Intelligence
• Nutritional Science

Background:

• Accurate dietary assessment is crucial for personalized health but challenged by complex meal presentations and limited culturally specific data.
• Traditional Turkish cuisine presents unique challenges for objective dietary evaluation due to its diverse and complex dishes.

Purpose of the Study:

• To develop and validate a confidence-aware hybrid vision-language framework for recognizing traditional Turkish foods.
• To enable structured nutritional assessment, including portion weight and energy estimation, for personalized health management.

Main Methods:

• Curated a dataset of 14,711 images across 40 Turkish culinary classes.
• Trained and evaluated seven deep learning architectures, with EfficientNet V2-L showing top performance.
• Implemented a confidence-aware routing strategy to leverage Google Gemini 2.5 Flash multimodal large language model (MLLM) for uncertain predictions.

Main Results:

• EfficientNet V2-L achieved 93.47% accuracy in standalone food recognition.
• The hybrid framework reached a combined classification accuracy of 95.50%.
• Portion weight and total energy were validated with a mean absolute error (MAE) of 18.42 g and 36.75 kcal, respectively.

Conclusions:

• The hybrid vision-language approach effectively addresses challenges in regional dietary intake evaluation.
• The developed framework, implemented as a mobile application, offers localized plate detection, portion analysis, and nutrient tracking.
• This scalable design supports consumer-facing digital nutrition platforms for enhanced health management.