Pic2Plate: A Vision-Language and Retrieval-Augmented Framework for Personalized Recipe Recommendations

Area of Science:

• Computer Science
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Personalized recipe recommendations are crucial for healthy eating but often lack accuracy in ingredient recognition and personalization.
• Existing systems struggle with integrating user preferences, dietary needs, and real-time ingredient availability.

Purpose of the Study:

• To introduce Pic2Plate, a novel framework for personalized recipe recommendations using Vision-Language Models (VLMs) and Retrieval-Augmented Generation (RAG).
• To enhance user experience in cooking by accurately identifying ingredients from images and providing tailored recipe suggestions.

Main Methods:

• Developed Pic2Plate, integrating VLMs for image-based ingredient detection and RAG for personalized recipe retrieval.
• Utilized smartphone camera sensors for accessible ingredient capture and GPT-4o for ingredient recognition.
• Evaluated ingredient detection using precision, recall, accuracy, and F1-score, and recipe relevance via a user survey and Bradley-Terry model.

Main Results:

• The ingredient detection module achieved high performance metrics: precision (0.83), recall (0.91), accuracy (0.77), and F1-score (0.86).
• Pic2Plate's VLM and RAG combination demonstrated superior recipe relevance compared to other models in user evaluations.
• The system effectively provides context-aware, reliable, and diverse recipe suggestions.

Conclusions:

• Pic2Plate offers a scalable and user-centric approach to transforming personalized cooking and recipe recommendation systems.
• The integration of image recognition and advanced AI models significantly improves the accuracy and relevance of recipe suggestions.
• This framework has the potential to make personalized nutrition more accessible and practical for everyday users.