Using Large Language Models to Automate the Comparison and Integration of Evolving Clinical Practice Guidelines into...

Clinical decision support systems (CDSSs) can improve the compliance of therapeutic decisions with clinical practice guidelines (CPGs). However, the rapid evolution of CPGs creates major challenges in keeping CDSS knowledge bases up to date. This study investigates the use of Large Language Models (LLMs) to automate the comparison of breast cancer CPGs over time, focusing on AP-HP (2016), implemented in the DESIREE guideline-based CDSS, and SENORIF (2021-2022). The aim was to automatically identify guideline-derived recommendations in both CPGs and update DESIREE knowledge base accordingly. The top-performing LLM, a fine-tuned Mistral Large model, achieved an average performance (F1 score: 0.49), and many inconsistencies were observed. While fine-tuned LLMs show promising potential for supporting guideline maintenance, they still face significant limitations, highlighting the need for structured CPG formats to ensure reliable CDSS integration.