Related Experiment Video
Updated: Jul 12, 2026
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Evaluation of Large Language Models in the Clinical Management of Patients With Upper Gastrointestinal Bleeding:
Mohsen Rajabnia1, Farbod Davoodi2, Mahsa Hajisafarali1
1Department of Medicine Alborz University of Medical Sciences Karaj Iran.
Objective:
To evaluate large language models (LLMs) for pre-endoscopy (PE) risk stratification and prediction of endoscopic findings in upper gastrointestinal bleeding (UGIB), and compare their performance with clinical risk scores, conventional machine learning (ML) models, and hybrid approaches.
Methods:
This multicenter retrospective study included 384 patients with UGIB who underwent endoscopy. Five LLMs (GPT-5, Gemini-2.5-Flash, Llama 4, Grok, and DeepSeek R1) were tested using structured zero-shot prompts based on PE clinical and laboratory data. Their performance in identifying high-risk patients was compared with the Glasgow-Blatchford Score (GBS), AIMS65, PE Rockall score, and ML models. Hybrid LLM-score models were also assessed. Two gastroenterologists evaluated the quality of LLM-generated justifications.
Results:
GBS showed the best discriminative performance among clinical scores (area under the receiver operating characteristic curve [AUROC] 0.73). Among LLMs, GPT-5 achieved the highest accuracy (0.66), while Grok showed the best-balanced performance (0.59; F1 0.47). Gemini-2.5-Flash had the highest sensitivity (0.89) but low specificity (0.21). Endoscopic prediction performance was modest, with Gemini-2.5-Flash achieving the highest exact-match accuracy (0.34) and micro-F1 (0.38). Hybrid models improved performance over standalone LLMs but did not outperform GBS alone (best: GBS+GPT-5, AUROC 0.670. LLMs showed higher numerical performance than conventional ML models, but a statistical comparison was not possible due to unavailable instance-level data. Grok received the highest human evaluation score for explanation quality.
Conclusions:
LLMs showed moderate performance in UGIB risk stratification and endoscopic prediction but were inferior to clinical scores, especially GBS. Hybrid models modestly improved over standalone LLMs but not GBS, supporting their use as adjunct tools rather than clinical decision-support systems.
Trial Registration:
N/A.
Related Concept Videos
01:28Esophageal Varices-II: Clinical Features and Management
In the initial assessment, a thorough review of the patient's medical history is vital to identify risk factors such as liver disease, alcohol abuse, or...
01:28Peptic Ulcer Disease III: Clinical Manifestations and Diagnostic Studies
Few clinical manifestations differentiate gastric ulcers from duodenal ulcers. Distinctions in the location, timing, and pain relief are crucial for healthcare providers in differentiating between gastric and duodenal ulcers during clinical assessments.
01:29Gastroesophageal Reflux Disease II: Clinical Features and Management
Clinical Manifestations
GERD presents itself in a multitude of ways, with symptoms varying from person to person. The hallmark symptoms are...
01:25Peptic Ulcer Disease III: Clinical Manifestations and Complications
01:26Peptic Ulcer Disease IV: Management
The therapeutic approach involves ensuring adequate rest, implementing drug therapy, promoting smoking cessation, making dietary modifications, and emphasizing long-term follow-up care.
Pharmacological management
The prevailing therapy for peptic ulcers involves a combination of managing the patient's current medication...