Related Experiment Video
Updated: Jul 16, 2026
03:14
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Fine-Tuning, Retrieval-Augmented Generation, and Hybrid Large Language Models for Postoperative Decision Support:
Srinivasagam Prabha1, Bernardo Gabriele Collaco1, Cesar Abraham Gomez-Cabello1
1Division of Plastic Surgery, Mayo Clinic in Florida, 4500 San Pablo Road South, Jacksonville, FL, 32224, United States, 1 904-953-2000.
Journal of Medical Internet Research
|July 15, 2026
Summary
Integrating medical knowledge into large language models (LLMs) significantly improves postoperative decision support. The hybrid fine-tuning and retrieval-augmented generation (RAG) approach demonstrated the highest accuracy and safety for patient education.
Area of Science:
- Artificial Intelligence in Medicine
- Clinical Decision Support Systems
- Natural Language Processing
Background:
- Large language models (LLMs) offer potential for clinical decision support but face challenges in integrating medical knowledge for accuracy and safety.
- Existing methods like fine-tuning and retrieval-augmented generation (RAG) have not been systematically compared for postoperative care.
Purpose of the Study:
- To compare the performance, reliability, and safety of baseline, fine-tuning, RAG, and hybrid fine-tuning+RAG LLM configurations.
- To evaluate LLM effectiveness for postoperative decision support, including discharge instructions and patient education.
Main Methods:
- A comparative evaluation of four LLM configurations (baseline, fine-tuning, RAG, fine-tuning+RAG) using Google Gemini 2.5 Flash.
- Model adaptation and validation using 600 postoperative Q&A pairs, with final evaluation on 150 diverse queries.
- Independent assessment by 3 clinical experts for accuracy, safety, completeness, and relevance, alongside automated metrics for readability and hallucination.
Main Results:
- All knowledge-enhanced LLMs significantly outperformed the baseline model in overall accuracy (e.g., fine-tuning+RAG achieved 97.3%).
- The fine-tuning+RAG configuration showed the highest clinical accuracy (96.7%) for in-scope queries and superior classification performance (100% precision, 96.7% recall).
- Knowledge-enhanced models improved safety/refusal accuracy, though readability was slightly reduced, partly due to standardized safety information.
Conclusions:
- Incorporating domain-specific knowledge via fine-tuning, RAG, or hybrid approaches enhances LLM performance for postoperative decision support.
- The fine-tuning+RAG model demonstrated the most favorable outcomes, suggesting its promise for patient education and clinical guidance.
- Further validation, readability optimization, and human oversight are crucial before patient-facing deployment of these enhanced LLMs.