Utility of OpenEvidence and ChatGPT in Selecting Neuroimaging Based on Appropriateness Criteria

Large Language models (LLMs) are novel tools with promising applications in imaging selection for clinical scenarios, but information is limited on the accuracy of an LLM trained on a medically specific dataset compared to a generalized dataset. In this analysis, we assessed the accuracy of the imaging recommendations by LLMs ChatGPT-3.5, ChatGPT-5.0, and OpenEvidence for neuroradiology, using the American College of Radiology (ACR) Appropriateness Criteria as the reference standard.

Prompts from the ACR Appropriateness Criteria priority clinical areas within neurology were used to build open-ended (OE) or select-all-that-apply (SATA) questions. Each question was posed three times, and the responses were scored against ACR guidelines. The performance of each LLM on OE and SATA was compared, and then was further analyzed by acuity. Mean accuracy scores were calculated by averaging performance across OE and SATA questions for each model. Analysis of variance was then conducted to assess differences in performance among the models.

A total of 78 prompts were assessed in 11 priority clinical areas. These areas included cerebrovascular conditions, neurological pain, and spine pathologies. Amongst SATA, Open Evidence performed the best (79.5%) compared to GPT5 (71.2%) and GPT3.5 (44.4%). For the OE prompts, Open Evidence was more accurate (86.3%) than GPT3.5 (69.2%) but not GPT5 (80.6%). Amongst the models, no differences were found based on acuity.

Overall, OpenEvidence performed the best for both OE and SATA prompts. ChatGPT-5 performed better than ChatGPT3.5 for SATA prompts. LLM performance was unaffected by the acuity of the case.