Artificial Intelligence for Patient Support: Assessing Retrieval-Augmented Generation for Answering Postoperative...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Plastic Surgery

Background:

• Large language models (LLMs) in healthcare risk patient safety due to inaccurate information.
• Retrieval-Augmented Generation (RAG) enhances AI reliability using curated knowledge bases.
• RAG is particularly beneficial for high-demand medical specialties like plastic surgery.

Purpose of the Study:

• To evaluate Retrieval-Augmented Generation (RAG) AI models for postoperative rhinoplasty patient inquiries.
• To assess the safety and identify areas for improvement in RAG model implementation for clinical use.

Main Methods:

• Four RAG models (Gemini-1.0-Pro-002, Gemini-1.5-Flash-001, Gemini-1.5-Pro-001, PaLM 2) were tested.
• Models answered 30 common patient questions using authoritative rhinoplasty texts.
• Responses were evaluated for accuracy, comprehensiveness, readability, and actionability using established metrics and statistical analysis.

Main Results:

• Responses were generally accurate (41.7% fully accurate), but a significant nonresponse rate (30.8%) was observed.
• Gemini-1.0-Pro-002 showed superior comprehensiveness, but readability and understandability were below patient education standards.
• PaLM 2 performed lowest in response actionability.

Conclusions:

• RAG application in rhinoplasty care demonstrates potential for accuracy but faces limitations in nonresponse and contextual understanding.
• Addressing these challenges is crucial for safe and effective RAG implementation in diverse medical fields.
• RAG models can potentially transform patient care by reducing physician workload and improving patient engagement.