Integrating clinical guidelines with large language models for improved sepsis mortality prediction

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Decision Support Systems
• Machine Learning for Healthcare

Background:

• Sepsis mortality prediction remains a critical challenge in intensive care units.
• Existing predictive models often lack integration with up-to-date clinical guidelines.
• Large language models (LLMs) offer potential for complex data analysis in healthcare.

Purpose of the Study:

• To develop and validate a clinical guideline-integrated LLM for enhanced sepsis mortality prediction.
• To evaluate the performance of the guideline-enhanced LLM against traditional and deep learning methods.
• To assess the impact of explicit clinical guideline integration on LLM performance.

Main Methods:

• Fine-tuning a large language model using Low-Rank Adaptation on MIMIC-IV data from 24,237 ICU sepsis patients.
• Embedding clinical guidelines directly into the LLM training process.
• Evaluating model performance using accuracy, F1-score, sensitivity, specificity, and AUC; conducting ablation studies.

Main Results:

• The guideline-enhanced LLM achieved superior performance: accuracy (0.819), F1-score (0.815), sensitivity (0.815), specificity (0.822), and AUC (0.852).
• Performance gains were notably higher than traditional machine learning (accuracy: 0.774, AUC: 0.850) and deep learning methods (accuracy: 0.762, AUC: 0.841).
• Ablation studies confirmed that explicit guideline integration significantly improved performance over direct prompting and fine-tuning without guidelines.

Conclusions:

• Incorporating clinical guidelines into LLM fine-tuning enhances sepsis mortality prediction robustness and accuracy.
• Guideline-integrated LLMs outperform existing machine learning and deep learning approaches for sepsis prediction.
• Explicit domain knowledge integration is crucial for developing reliable clinical AI tools.