Development and prospective shadow evaluation of a domain-specific large language model for emergency neurological...

Large language models (LLMs) show promise in emergency medicine; however, their role in emergency neurology remains unclear. We developed a customized LLM, Xuanwu-NeuroAid, and prospectively enrolled 433 patients. The diagnostic outputs of the model and emergency physicians were compared with confirmed diagnoses, and an expert panel performed blinded evaluations of the recommendations generated by both the model and physicians. The independent diagnostic accuracy of the model was 79.4%, which was significantly higher than that of emergency physicians (65.4%, p < 0.001). Blinded expert assessments indicated that the model's recommendations for examinations and treatments were significantly more comprehensive, accurate, and clinically applicable than those of physicians (p < 0.001). Moreover, the inclusion of demographic information altered the model's recommendations for health education, suggesting a sensitivity to sociodemographic factors. Our findings highlight the potential of the LLM to enhance diagnostic precision and support decision-making in emergency neurology under simulated clinical conditions. This study was registered at ClinicalTrials.gov (NCT06779292; January 6, 2025).