Related Experiment Video
Updated: May 22, 2025
A Postoperative Evaluation Guideline for Computer-Assisted Reconstruction of the Mandible
Published on: January 28, 2020
Artificial Intelligence for Patient Support: Assessing Retrieval-Augmented Generation for Answering Postoperative
Background:
Although artificial intelligence (AI) is revolutionizing healthcare, inaccurate or incomplete information from pretrained large language models (LLMs) like ChatGPT poses significant risks to patient safety. Retrieval-augmented generation (RAG) offers a promising solution by leveraging curated knowledge bases to enhance accuracy and reliability, especially in high-demand specialties like plastic surgery.
Objectives:
This study evaluates the performance of RAG-enabled AI models in addressing postoperative rhinoplasty questions, aiming to assess their safety and identify necessary improvements for effective implementation into clinical care.
Methods:
Four RAG models (Gemini-1.0-Pro-002, Gemini-1.5-Flash-001, Gemini-1.5-Pro-001, and PaLM 2) were tested on 30 common patient inquiries. Responses, sourced from authoritative rhinoplasty texts, were evaluated for accuracy (1-5 scale), comprehensiveness (1-3 scale), readability (Flesch Reading Ease [FRE], Flesch-Kincaid Grade Level), and understandability/actionability (Patient Education Materials Assessment Tool). Statistical analyses included Wilcoxon rank sum, Armitage trend tests, and pairwise comparisons.
Results:
When responses were generated, they were generally accurate (41.7% completely accurate); however, a 30.8% nonresponse rate revealed potential challenges with query context interpretation and retrieval. Gemini-1.0-Pro-002 demonstrated superior comprehensiveness (P < .001), but readability (FRE: 40-49) and understandability (mean: 0.7) fell below patient education standards. PaLM 2 scored lowest in actionability (P < .007).
Conclusions:
This first application of RAG to postoperative rhinoplasty patient care highlights its strengths in accuracy alongside its limitations, including nonresponse and contextual understanding. Addressing these challenges will enable safer, more effective implementation of RAG models across diverse surgical and medical contexts, with the potential to revolutionize patient care by reducing physician workload while enhancing patient engagement.
