Performance of Vision-Enabled Large Language Models in Image-Based Electrocardiogram Interpretation: Exploratory...

Area of Science:

• Artificial Intelligence in Medicine
• Medical Image Analysis
• Cardiology

Background:

• Vision-enabled large language models (VE-LLMs) offer potential for flexible and explainable medical image interpretation.
• Systematic assessment of VE-LLM performance on real-world clinical data like 12-lead electrocardiograms (ECGs) is lacking.

Purpose of the Study:

• Evaluate the diagnostic accuracy and reliability of generalist VE-LLMs for interpreting real-world ECG images.
• Compare the performance of generalist VE-LLMs against specialized ECG LLMs.

Main Methods:

• Tested 6 generalist VE-LLMs and 2 specialized ECG LLMs on 70 deidentified ECG images.
• Standardized prompts requested 9 ECG determinations (rhythm, blocks, QTc, etc.).
• Expert consensus served as the reference standard; outputs evaluated using diagnostic metrics.

Main Results:

• Overall balanced accuracy for generalist VE-LLMs ranged from 50.1% to 61.8%.
• Sensitivity for key findings like atrial fibrillation, AV block, and ST-segment deviation was notably low across generalist models.
• Agreement with expert interpretation was poor-to-fair (Cohen κ ≤0.39). Specialized models showed improved performance in specific tasks.

Conclusions:

• Generalist VE-LLMs demonstrate moderate overall performance but insufficient sensitivity and expert agreement for clinical ECG interpretation.
• Performance is inconsistent across models and diagnostic categories, precluding current clinical deployment.
• Specialized ECG LLMs show promise but require further validation.