Building a Silver-Standard Dataset from NICE Guidelines for Clinical LLMs

Large language models (LLMs) are increasingly used in healthcare, yet standardised benchmarks for evaluating guideline-based clinical reasoning are missing. This study introduces a validated dataset derived from the UK's National Institute for Health and Care Excellence (NICE) guidelines covering multiple diagnoses. The dataset was initially created with the help of GPT-4o-mini and contains patient scenarios, as well as clinical questions; notably, generated contents were further assessed for clinical validity and realism by a practicing clinician to ensure alignment with NICE guidelines. We benchmark a range of recent popular LLMs to showcase the validity of our NICE guideline-based dataset. The framework supports systematic evaluation of LLMs' clinical utility and guideline adherence. The dataset and Appendix are publicly available at https://github.com/julia-ive/guidelines_qa.