Real-Time Evaluation of a Large Language Model for Clinical Practice Guideline Development

Area of Science:

• Artificial Intelligence in Healthcare
• Clinical Practice Guideline Development
• Health Informatics

Background:

• Clinical practice guideline development is a complex, multi-step process.
• Large language models (LLMs) are emerging tools with potential applications in healthcare.
• Evaluating LLM capabilities in guideline development is crucial for understanding their utility.

Purpose of the Study:

• To assess the effectiveness of a large language model (LLM) in executing all phases of clinical practice guideline development.
• To evaluate the performance of OpenAI's Generative Pretrained Transformer (GPT)-4o in guideline creation.

Main Methods:

• The study utilized GPT-4o to perform tasks aligned with the Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) framework.
• LLM prompts were executed concurrently with the guideline panel's progress on a guideline for neuromuscular blockade in acute respiratory distress syndrome.
• The evaluation covered question framing, outcome selection/rating, evidence summarization, quality assessment, and evidence-to-decision frameworks.

Main Results:

• The LLM demonstrated significant utility in the initial stages, particularly in framing healthcare questions and selecting/rating outcomes.
• The LLM's limitations became more evident in subsequent steps, including evidence summarization and quality assessment.
• The evidence-to-decision framework creation highlighted the LLM's challenges in synthesizing complex evidence.

Conclusions:

• LLMs show the most promise in the early, foundational steps of clinical practice guideline development.
• Current LLM capabilities are limited in the more intricate phases of evidence synthesis and recommendation formulation.
• Further development and validation are required to enhance LLM performance across the entire guideline development lifecycle.