Performance Modeling of Lightweight Retrieval-Augmented Large Language Models for Low-Resource Plastic Surgery...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine
• Surgical Education Technology

Background:

• Large language models (LLMs) offer potential for surgical education but face challenges with accuracy and reliability.
• Retrieval-augmented generation (RAG) aims to improve LLM accuracy by grounding responses in external knowledge bases.
• The performance impact of various RAG configurations, especially concerning computational costs, is not well understood.

Purpose of the Study:

• To evaluate the performance of lightweight, open-source Retrieval-Augmented Generation (RAG) configurations for plastic surgery applications.
• To identify key factors influencing RAG performance across different query complexities and subspecialties.
• To determine optimal RAG configurations for reliable LLM-assisted surgical reference.

Main Methods:

• Evaluated 120 lightweight, open-source RAG configurations using 40 plastic surgery QA tasks (single- and multi-hop).
• Varied base LLMs (Phi-3-mini-128k-instruct, BioMistral-7B), embedding models, database sizes, and chunk sizes.
• Assessed performance via semantic similarity (Ragas) to physician-validated answers, analyzed using linear mixed-effects regression.

Main Results:

• Lightweight, open-source RAG models can achieve high performance.
• Phi-3-mini-128k-instruct showed consistent performance across query types, while BioMistral-7B excelled in specific configurations.
• Larger database sizes and specific embedding models (bge-large-en-v1.5) improved performance; multi-hop queries decreased it, though less so for Phi-3-mini-128k-instruct.

Conclusions:

• Effective RAG systems for plastic surgery do not necessitate large proprietary models; performance hinges on configuration choices.
• Interaction effects between components significantly influence RAG system outcomes.
• Future advancements in predictive modeling could facilitate resource-efficient and safe clinical RAG deployment.