CPGPrompt: translating clinical guidelines into large language model-executable decision support

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Decision Support Systems
• Natural Language Processing

Background:

• Integrating clinical practice guidelines (CPGs) into artificial intelligence (AI) is challenging due to limitations in existing methods.
• Previous AI approaches like rule-based systems or black-box models lack interpretability and domain applicability.

Purpose of the Study:

• To develop and validate CPGPrompt, an auto-prompting system that converts narrative CPGs into large language models (LLMs).
• To address the limitations of current AI integration of CPGs for improved patient care.

Main Methods:

• The CPGPrompt framework translates CPGs into structured decision trees.
• A large language model (LLM) dynamically navigates these trees for patient case evaluation.
• Synthetic vignettes across headache, lower back pain, and prostate cancer domains were used for testing.

Main Results:

• CPGPrompt achieved strong performance in binary specialty referral classification (F1: 0.85-1.00) across all tested domains.
• Multiclass pathway assignment performance varied by domain: headache (F1: 0.47), lower back pain (F1: 0.72), and prostate cancer (F1: 0.77).
• Performance differences were linked to guideline structure, negation handling, temporal reasoning needs, and reliance on quantifiable data.

Conclusions:

• CPGPrompt demonstrates generalizability and high sensitivity for referral decisions, offering advantages over black-box AI.
• The system's transparent framework aids in identifying failure modes.
• Further improvements are needed for handling subjective clinical assessments to enhance clinical robustness.