Using Large Language Models to Automate the Comparison and Integration of Evolving Clinical Practice Guidelines into...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Practice Guidelines

Background:

• Clinical decision support systems (CDSSs) enhance adherence to clinical practice guidelines (CPGs).
• Rapid updates in CPGs pose challenges for maintaining CDSS knowledge bases.
• Automating CPG comparison is crucial for efficient CDSS maintenance.

Purpose of the Study:

• To investigate the use of Large Language Models (LLMs) for automating the comparison of breast cancer CPGs over time.
• To identify guideline-derived recommendations and update the DESIREE CDSS knowledge base.
• To assess the performance of LLMs in detecting inconsistencies between guideline versions.

Main Methods:

• Comparison of two breast cancer CPGs: AP-HP (2016) and SENORIF (2021-2022).
• Utilized fine-tuned Large Language Models (LLMs), including Mistral Large, for automated recommendation extraction and comparison.
• Evaluated LLM performance using F1 score to measure accuracy in identifying guideline-derived recommendations.

Main Results:

• The top-performing LLM (fine-tuned Mistral Large) achieved an average F1 score of 0.49.
• Numerous inconsistencies were identified between the compared breast cancer CPG versions.
• LLM performance indicates potential but also highlights significant limitations in current capabilities.

Conclusions:

• Fine-tuned LLMs offer promising potential for automating the maintenance of CDSS knowledge bases.
• Significant limitations exist in current LLM performance for CPG comparison and recommendation extraction.
• Structured CPG formats are essential for ensuring reliable integration and updating of CDSSs.