Assessing ChatGPT4 with and without retrieval-augmented generation in anticoagulation management for gastrointestinal...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine
• Gastroenterology

Background:

• Managing anticoagulation for gastrointestinal (GI) procedures is increasingly complex.
• This study addresses the need for accurate medical guidance in this area.

Purpose of the Study:

• To evaluate ChatGPT-4's accuracy in providing medical guidance for anticoagulation management during GI procedures.
• To compare ChatGPT-4 with earlier AI models (ChatGPT-3.5) and a retrieval-augmented generation model (ChatGPT4-RAG).

Main Methods:

• Thirty-six anticoagulation-related questions based on professional guidelines were posed to ChatGPT-4.
• Nine gastroenterologists assessed the accuracy and relevance of AI-generated responses.
• ChatGPT-4's performance was benchmarked against ChatGPT-3.5 and ChatGPT4-RAG, with a survey on clinician perceptions.

Main Results:

• ChatGPT-4 demonstrated improved accuracy over ChatGPT-3.5 (30.5% fully accurate, 47.2% generally accurate).
• ChatGPT4-RAG achieved 75% full accuracy, showing enhanced integration of current information.
• Specific procedure accuracy rates varied: esophagogastroduodenoscopy (51.8%), endoscopic retrograde cholangiopancreatography (42.8%), and colonoscopy (50%).

Conclusions:

• ChatGPT4-RAG offers advancements in anticoagulation management for endoscopic procedures but requires further validation.
• A 75% accuracy rate is insufficient for independent clinical decision-making due to medicolegal concerns.
• AI is best utilized to support, not replace, clinical judgment, emphasizing patient confidentiality and the physician-patient relationship.