Evaluating Large Language Models for Extracting Clinical Recommendations from Practice Guidelines: A Preliminary...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Knowledge Management

Background:

• Clinical Practice Guidelines (CPGs) are essential for evidence-based healthcare.
• Accessing and utilizing CPG content can be challenging for clinicians.
• Large Language Models (LLMs) offer potential for automating information extraction from complex documents.

Purpose of the Study:

• To evaluate the effectiveness of four different LLMs in extracting clinical recommendations from CPGs.
• To assess the ability of LLMs to categorize extracted recommendations.
• To compare LLM performance with and without an example set of extracted recommendations.

Main Methods:

• Four distinct LLMs were tested for their ability to extract and categorize recommendations from CPGs.
• Two testing conditions were employed: one with an example set and one without.
• Accuracy and completeness of extracted recommendations were key evaluation metrics.

Main Results:

• DeepSeek and Grok demonstrated superior performance among the tested LLMs.
• These models achieved over 90% accuracy in extracting clinical recommendations.
• The inclusion of an example set influenced the extraction and categorization process.

Conclusions:

• LLMs show significant potential for automating knowledge extraction from clinical guidelines.
• Preliminary findings highlight both the capabilities and limitations of current LLMs in this domain.
• Further research is needed to optimize LLM application for clinical knowledge management.