The Potential Clinical Utility of the Customized Large Language Model in Gastroenterology: A Pilot Study.

Area of Science:

• Gastroenterology
• Artificial Intelligence
• Clinical Decision Support

Background:

• Large language models (LLMs) have potential in clinical practice, but their application in gastroenterology is understudied.
• Gastroenterology presents unique challenges for AI due to complex, specialized information.

Purpose of the Study:

• To explore the clinical utility of a customized GPT model and a conventional GPT-4o in gastroenterology.
• To compare the performance of LLMs against a gastroenterology fellow and an expert gastroenterologist.

Main Methods:

• A customized GPT model using BM25 and GPT-4o was developed with internal medicine and Korean gastroenterology textbooks.
• A conventional ChatGPT 4o was accessed for comparison.
• A benchmark of 15 clinical questions, developed by experts, was used to test the LLMs, a fellow, and an expert.

Main Results:

• The customized LLM answered 8/15 questions correctly; its performance improved with English terminology but struggled with judgment-based questions.
• Conventional GPT-4o achieved the highest AI score (14/15), outperforming the customized model and a gastroenterology fellow (10/15).
• Both LLMs performed comparably to or better than the fellow, indicating significant potential, though slightly below the expert (15/15).

Conclusions:

• LLMs can assist with specialized tasks like patient counseling in gastroenterology.
• Retrieval-augmented generation (RAG) is crucial for handling complex, specialized medical data.
• Clinician oversight is essential for the safe and effective integration of LLMs into clinical practice.