Related Experiment Video
Updated: Jan 10, 2026
03:14
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
1000
The Development and Evaluation of a Retrieval-Augmented Generation Large Language Model Virtual Assistant for
Syed Ali Haider1, Srinivasagam Prabha1, Cesar Abraham Gomez Cabello1
1Division of Plastic Surgery, Mayo Clinic, Jacksonville, FL 32224, USA.
Bioengineering (Basel, Switzerland)
|November 27, 2025
Summary
The AI Virtual Assistant (AIVA) Platform enhances postoperative recovery by providing accurate, safe, and relevant patient guidance. This AI system combines NLP and LLMs to address information gaps and reduce healthcare overutilization.
Area of Science:
- Artificial Intelligence in Healthcare
- Natural Language Processing
- Large Language Models
Background:
- Postoperative recovery is hindered by patient information gaps, leading to provider burden and healthcare overutilization.
- Traditional discharge materials are often ineffective, contributing to patient overwhelm and unnecessary ER visits.
- Existing conversational AI, including Natural Language Processing (NLP) and Large Language Models (LLMs), faces limitations in accuracy and safety.
Purpose of the Study:
- To develop the AI Virtual Assistant (AIVA) Platform, integrating NLP and LLMs for dynamic postoperative guidance.
- To utilize a retrieval-augmented generation (RAG) architecture with Gemini 2.0 Flash and a verified medical knowledge base.
- To provide safe, patient-facing postoperative information grounded in validated clinical content.
Main Methods:
- The AIVA Platform was evaluated using 750 simulated patient interactions across 20 recovery domains.
- Physician reviewers assessed system performance, including classification metrics, relevance, completeness, and consistency.
- Automated metrics evaluated groundedness, fluency, and readability, while safety guardrails were tested with out-of-scope and emergency scenarios.
Main Results:
- The system achieved 98.4% classification accuracy with high physician-rated completeness (4.83/5) and relevance (SSI Index 2.68/3).
- Safety guardrails successfully identified 100% of out-of-scope and escalation scenarios.
- Groundedness evaluations showed strong precision (0.951) and faithfulness (0.956), with 95.6% verification agreement.
Conclusions:
- Simulated testing confirmed the AIVA Platform's technical accuracy, safety, and clinical relevance for postoperative care.
- The RAG architecture effectively balances the flexibility of LLMs with the safety of deterministic NLP.
- Findings support readiness for clinical trials, with readability identified as an area for future improvement.
