An AI-powered RAG Conversational Agent for Enhancing Patient Recall of Surgical Consultations: A Proof-of-concept...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Natural Language Processing

Background:

• Patient recall of dense surgical consultation information is challenging.
• Existing tools lack access to individualized clinical dialogue.
• Retrieval-Augmented Generation (RAG) can ground AI responses in source transcripts for patient recall.

Purpose of the Study:

• Evaluate the technical feasibility of a RAG conversational agent.
• Assess the performance of a RAG agent using synthetic plastic surgery encounters.

Main Methods:

• Generated 20 simulated plastic surgery cases with 4 visits each.
• Used Whisper for audio transcription and Vertex AI RAG with Gemini 2.0 Pro.
• Developed 200 patient questions and evaluated responses against gold standards for accuracy and recall.

Main Results:

• Achieved 99.0% accuracy, perfect precision, and 0.99 recall.
• Demonstrated appropriate uncertainty handling and no hallucinations.
• Readability scores indicated accessible patient-level language (Flesch-Kincaid Grade Level 8.56).

Conclusions:

• RAG agents grounded in consultation transcripts offer accurate, patient-friendly recall support.
• The system shows strong factual reliability and handles uncertainty appropriately.
• Findings support the feasibility of transcript-based patient assistance for future deployment.