Enhancing Large Language Models for Clinical Decision Support by Incorporating Clinical Practice Guidelines

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Informatics
• Natural Language Processing

Background:

• Clinical Decision Support (CDS) systems are crucial for evidence-based healthcare.
• Integrating Clinical Practice Guidelines (CPGs) into Large Language Models (LLMs) for CDS is an underexplored area.
• Existing LLM approaches for CDS lack systematic CPG integration.

Purpose of the Study:

• To develop and evaluate novel methods for incorporating CPGs into LLMs for enhanced CDS.
• To assess the performance of different CPG integration strategies using COVID-19 outpatient treatment as a case study.
• To compare LLM performance with and without CPG enhancement.

Main Methods:

• Developed three CPG incorporation methods: Binary Decision Tree (BDT), Program-Aided Graph Construction (PAGC), and Chain-of-Thought-Few-Shot Prompting (CoT-FSP).
• Utilized Zero-Shot Prompting (ZSP) as a baseline.
• Evaluated methods using synthetic patient data and four LLMs (GPT-4, GPT-3.5 Turbo, LLaMA, PaLM 2) via automatic and human assessments.

Main Results:

• All LLMs demonstrated improved performance when enhanced with CPGs compared to the baseline ZSP.
• BDT achieved superior results in automatic evaluations over CoT-FSP and PAGC.
• All proposed CPG integration methods showed high effectiveness in human evaluations.

Conclusions:

• LLMs augmented with CPGs significantly enhance the accuracy of clinical decision support for COVID-19 outpatient treatment.
• The developed methods, particularly BDT, offer effective strategies for integrating CPGs into LLMs.
• These findings suggest broad applicability of CPG-enhanced LLMs in various clinical decision-making scenarios.