GastroGPT: Development and controlled testing of a proof-of-concept customized clinical language model.

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Natural Language Processing
• Gastroenterology AI Applications

Background:

• General-purpose AI large language models (LLMs) have limited clinical utility, primarily used for documentation and summarization.
• Specialty-specific AI models are needed to enhance performance in complex medical domains.
• Gastroenterology presents a unique challenge for AI due to its diverse clinical tasks.

Purpose of the Study:

• To develop and evaluate GastroGPT, a novel, specialty-specific, multi-task clinical LLM for gastroenterology.
• To compare the performance of GastroGPT against leading general-purpose LLMs (GPT-4, Bard, Claude).
• To assess AI model efficacy across diverse gastroenterology case scenarios and clinical tasks.

Main Methods:

• A structured comparison of GastroGPT with three state-of-the-art general-purpose LLMs.
• Evaluation across seven key gastroenterology tasks and 10 simulated cases of varying complexity.
• Blinded expert panel assessment of clinical utility using a 10-point Likert scale and statistical analysis.

Main Results:

• GastroGPT achieved significantly higher overall scores (8.1) compared to GPT-4 (5.2), Bard (5.7), and Claude (7.0) (P < 0.001).
• GastroGPT outperformed general LLMs in six of seven clinical tasks and demonstrated superior score consistency.
• Model performance was consistent across case complexities for GastroGPT, unlike general models (P < 0.001).

Conclusions:

• GastroGPT demonstrates superior clinical utility and task performance compared to general-purpose LLMs in gastroenterology.
• Specialty-specific AI models offer significant advantages over general models for targeted medical applications.
• This study highlights the potential of tailored AI solutions to advance clinical medicine.