Evaluating large language models for diagnostic reasoning from unstructured clinical narratives in epilepsy

Area of Science:

• Artificial Intelligence
• Clinical Informatics
• Medical Diagnostics

Background:

• Large Language Models (LLMs) demonstrate capacity for encoding clinical knowledge.
• Existing evaluations often use structured data, neglecting unstructured clinical narratives.
• Real-world clinical settings present challenges in interpreting and reasoning with complex text.

Purpose of the Study:

• To evaluate the diagnostic performance of eight Large Language Models (LLMs) on an epilepsy seizure onset zone task.
• To assess the impact of prompt engineering strategies, including clinical role impersonation and context variations.
• To analyze the interpretability, reasoning quality, and source citation accuracy of LLM predictions.

Main Methods:

• Eight LLMs were tasked with mapping seizure descriptions to seven seizure onset zones.
• Quantitative and qualitative analyses measured correctness, confidence, calibration, and expert-evaluated reasoning.
• Systematic prompt-engineering and ablation studies explored performance dependencies.

Main Results:

• Prompt-engineered LLMs achieved above-chance accuracy, approaching clinician-level performance.
• Clinician-guided chain-of-thought reasoning significantly improved model performance.
• Model performance varied with clinical impersonation, narrative length, and language context.
• Expert review identified instances of correct predictions based on hallucinated knowledge and inaccurate citations.

Conclusions:

• SemioLLM offers a framework for evaluating LLMs in clinical domains using unstructured data.
• The study highlights both the potential and limitations of LLMs in healthcare applications.
• Improving LLM interpretability and source citation accuracy is crucial for clinical adoption.