Comparative Diagnostic Performance of a Multimodal Large Language Model Versus a Dedicated Electrocardiogram AI in...

Area of Science:

• Cardiology and Artificial Intelligence
• Medical Imaging Analysis
• Machine Learning in Healthcare

Background:

• Accurate electrocardiogram (ECG) interpretation is vital for emergency myocardial infarction (MI) diagnosis.
• Multimodal large language models (LLMs) show potential in clinical interpretation but their reliance on waveform data versus text cues is unclear.
• Direct comparisons between general LLMs and dedicated ECG AI tools are needed.

Purpose of the Study:

• To evaluate the diagnostic performance of ChatGPT and Gemini for MI detection from ECG images.
• To compare the performance of these general LLMs against a dedicated AI tool, ECG Buddy.

Main Methods:

• Retrospective analysis of a publicly available 12-lead ECG dataset from Pakistan (239 MI-positive, 689 MI-negative cases).
• ChatGPT (GPT-4o) and Gemini (2.5 Pro) were queried for MI confidence.
• ECG Buddy analyzed ECG images for ST-elevation MI, acute coronary syndrome, and myocardial injury biomarkers.

Main Results:

• ECG Buddy achieved high accuracy (96.98%), AUC (98.8%), sensitivity (96.65%), and specificity (97.10%).
• ChatGPT showed lower performance: accuracy 65.95%, AUC 57.34%, sensitivity 36.40%, specificity 76.2%.
• Gemini 2.5 Pro demonstrated poor performance: accuracy 29.63%, AUC 51.63%, sensitivity 97.07%, specificity 6.24%.
• LLM explanations for diagnoses were often inaccurate; ECG Buddy significantly outperformed both LLMs (P<.001).

Conclusions:

• General LLMs like ChatGPT and Gemini significantly underperform compared to specialized AI tools like ECG Buddy for ECG-based MI diagnosis.
• Domain-specific AI models are essential for achieving reliable and robust diagnostic accuracy in clinical settings.
• While LLMs may improve with further training, specialized AI currently remains critical for accurate MI detection from ECGs.