Supervised machine learning compared to large language models for identifying functional seizures from medical...

Area of Science:

• Neurology
• Artificial Intelligence
• Machine Learning

Background:

• The Functional Seizures Likelihood Score (FSLS) is a supervised machine learning tool designed to distinguish functional seizures (FS) from epileptic seizures (ES).
• Large Language Models (LLMs) possess the capability to identify complex patterns in data without specific prior training.

Purpose of the Study:

• To compare the diagnostic performance of the FSLS against two LLMs, ChatGPT and GPT-4, in differentiating FS from ES.
• To evaluate the relative strengths and weaknesses of a targeted machine learning approach versus a general LLM approach in seizure diagnosis.

Main Methods:

• 114 anonymized patient cases (FS, ES, mixed, or physiologic seizure-like events) were utilized.
• Text-based clinical data, including history of present illness, EEG, and neuroimaging results, were presented to LLMs in sequential prompts.
• Diagnostic accuracy and area under the receiver-operating characteristic (ROC) curves (AUCs) were compared between FSLS, ChatGPT, and GPT-4 using statistical analysis.

Main Results:

• GPT-4 achieved higher diagnostic accuracy (85%) and AUC (87%) compared to FSLS (74% accuracy, 85% AUC) and ChatGPT.
• Agreement between FSLS and GPT-4 was fair (Cohen's kappa = 40%).
• LLMs exhibited inconsistency, providing different predictions for the same patient data on separate occasions (33% of cases), with GPT-4's self-rated certainty correlating with this variability.

Conclusions:

• Both GPT-4 and FSLS can identify a significant number of patients with FS based on clinical history.
• The differing prediction patterns suggest LLMs utilize distinct identification methods compared to structured scores like FSLS.
• The observed inconsistency and lack of self-awareness regarding variability in LLM predictions warrant caution for their integration into clinical practice for FS diagnosis.