Large Language Model - Enhanced Decision Tree Framework for Identifying Multiple Sclerosis Diagnoses from Clinical...

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Decision Support Systems
• Natural Language Processing for Healthcare

Background:

• Early diagnosis of multiple sclerosis (MS) is critical but often delayed.
• Large language models (LLMs) show potential in extracting diagnostic information from clinical text.
• Generative pre-trained transformers (GPTs) can aid in streamlining diagnostic workflows.

Purpose of the Study:

• To develop a computable algorithm using GPT-4 to determine MS diagnosis status from the first neurology note.
• To apply the 2017 McDonald criteria within a structured decision framework for AI-driven reasoning.
• To assess the feasibility of using LLMs for early identification of MS.

Main Methods:

• Analysis of 125 first neurology notes from patients with MS, related disorders, and controls.
• Utilized clinical history and diagnostic testing sections, redacting assessment and plan.
• Converted 2017 McDonald criteria into a decision tree, guiding GPT-4 with expert knowledge for node-level reasoning.
• Evaluated GPT-4 performance against neurologist diagnoses and characterized AI hallucinations.

Main Results:

• GPT-4 achieved 84% accuracy, 79% precision, 74% recall, and 91% specificity in predicting MS diagnosis from initial neurology notes.
• The study cohort was representative (mean age 40±13 years; 81% women).
• Hallucinations were observed in 26% of cases, primarily incoherence and overreliance on input.

Conclusions:

• A structured, LLM-guided framework can identify potential MS diagnoses from early clinical documentation.
• Further large-scale studies are necessary to mitigate AI hallucinations and validate the approach.
• Implementation in clinical settings requires further testing and refinement.