ChatGPT's Limitations in Athlete ECG Interpretation: Evidence from a Multicenter Diagnostic Study

Area of Science:

• Cardiology
• Artificial Intelligence
• Medical Diagnostics

Background:

• Artificial intelligence (AI) shows promise in electrocardiogram (ECG) interpretation via deep learning.
• Large language models (LLMs) are increasingly explored for clinical applications, including ECG analysis.
• The diagnostic utility of general-purpose LLMs for differentiating cardiovascular disease from athlete ECGs during pre-participation screening (PPS) is unknown.

Purpose of the Study:

• To evaluate the diagnostic performance of a general-purpose LLM in identifying cardiovascular disease in competitive athletes undergoing PPS.
• To assess the LLM's ability to discriminate between diseased and non-diseased athletes based on ECG findings.

Main Methods:

• A multicentre diagnostic accuracy study involving 2950 athletes undergoing PPS.
• Evaluation of a commercial LLM (ChatGPT, version 5) using resting 12-lead ECGs.
• Reference standard: confirmed cardiovascular disease after full diagnostic work-up (n=450). LLM generated a disease likelihood score (0-100) without fine-tuning.

Main Results:

• LLM scores exhibited a significant floor effect (median 0) with substantial overlap between diseased and non-diseased groups.
• Receiver operating characteristic (ROC) analysis revealed an area under the curve of 0.52, indicating performance near random classification.
• At the optimal threshold, 79% of athletes with confirmed disease were misclassified as negative, with missed borderline and red-flag ECG abnormalities.

Conclusions:

• A general-purpose LLM, used without task-specific training, demonstrated no clinically meaningful diagnostic ability for cardiovascular disease in this athlete cohort.
• The findings suggest that current general-purpose LLMs are unsuitable for diagnostic triage in athlete screening without domain adaptation.
• Further research into specialized AI models or LLM fine-tuning is needed for reliable ECG interpretation in this context.