Artificial intelligence performance in clinical neurology queries: the ChatGPT model

Area of Science:

• Biomedical Literature
• Artificial Intelligence in Healthcare

Background:

• Artificial intelligence (AI) is rapidly advancing in biomedical fields.
• ChatGPT has become a widely adopted AI application.
• Evaluating AI tools for medical applications is crucial.

Purpose of the Study:

• To assess the accuracy and comprehensiveness of ChatGPT responses in clinical neurology.
• To identify AI performance variations across different neurological subspecialties and question types.

Main Methods:

• Administered 216 clinical neurology questions to ChatGPT.
• Categorized questions into multiple-choice, descriptive, and binary formats.
• Assessed question difficulty and the need for clinical reasoning and interpretation.

Main Results:

• ChatGPT achieved an overall accuracy rate of 65.3%.
• No significant differences in accuracy were found based on question style or difficulty.
• Accuracy and comprehensiveness were significantly lower for questions requiring critical interpretation (p < 0.01).

Conclusions:

• ChatGPT demonstrates moderate performance in clinical neurology but is inadequate for critical interpretation tasks.
• AI tools require careful validation for medical use, especially in specialized fields.
• Clinicians must verify AI-generated medical information using reliable sources.