Artificial intelligence versus neurologists: A comparative study on multiple sclerosis expertise

Multiple sclerosis (MS) is an autoimmune neurodegenerative disease affecting the central nervous system. MS diagnosis is complex, requiring magnetic resonance imaging and cerebrospinal fluid analysis due to the lack of definitive biomarkers. Although treatment advancements have reduced disability, diagnostic and therapeutic challenges remain, even among MS-specialized neurologists. Artificial Intelligence (AI) tools, which analyze large datasets, are increasingly used in healthcare, especially for diagnosis and treatment.

This study aims to assess the accuracy and scope of knowledge regarding MS, focusing on diagnosis, treatment options and management strategies, as tested among neurologists and AI bots.

Twenty multiple-choice questions, developed by MS-experienced neurology academics, were administered to 37 neurology specialists and 79 neurology residents in Turkey. The same questions were posed to AI platforms, including ChatGPT-4.0, GPT-4o, Gemini 1.5 Pro, Claude 3.5, Perplexity, and Perplexity Pro.

Neurology specialists answered 12.05 ± 4.01 questions correctly on average, while residents scored 9.08 ± 3.41. Among residents with more than two years of training, the correct answer rate improved to 11.96 ± 3.5. Specialists active in MS clinics scored significantly higher than other neurologists (17.67 ± 1.75). AI platforms scored between 14 and 19 out of 20; with an average of 17.0 ± 1.79 with Claude 3.5 scoring highest.

The findings suggest AI holds promise in supporting MS diagnosis and treatment, though challenges remain in nuanced cases. While AI complements neurologists, further studies are essential to understand its potential and limitations.

Aggregated data will be shared upon written request to the corresponding author.