Building a Silver-Standard Dataset from NICE Guidelines for Clinical LLMs

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Decision Support

Background:

• Large language models (LLMs) are increasingly adopted in healthcare settings.
• Standardized benchmarks for evaluating LLM-based clinical reasoning are currently lacking.
• Existing LLM evaluations often do not focus on adherence to clinical guidelines.

Purpose of the Study:

• To introduce a novel, validated dataset for evaluating LLMs in healthcare.
• To assess the clinical reasoning and guideline adherence of popular LLMs.
• To provide a framework for systematic evaluation of LLMs in clinical contexts.

Main Methods:

• Dataset creation using GPT-4o-mini based on UK National Institute for Health and Care Excellence (NICE) guidelines.
• Inclusion of diverse patient scenarios and clinical questions across multiple diagnoses.
• Clinical validation and realism assessment by a practicing clinician to ensure alignment with NICE guidelines.
• Benchmarking of several recent LLMs using the developed dataset.

Main Results:

• Demonstration of the dataset's validity for benchmarking LLMs.
• Showcasing the performance of various LLMs on guideline-based clinical reasoning tasks.
• Highlighting the utility of the framework for systematic LLM evaluation.

Conclusions:

• The developed NICE guideline-based dataset provides a crucial resource for evaluating LLMs in healthcare.
• The framework enables systematic assessment of LLM clinical utility and adherence to established guidelines.
• Public availability of the dataset facilitates further research and development in AI-driven healthcare.