Performance of large language models in electrocardiogram interpretation: A comparative study

Area of Science:

• Artificial Intelligence in Medicine
• Cardiology
• Machine Learning for Healthcare

Background:

• Large language models (LLMs) are increasingly utilized for medical interpretations.
• Evaluating LLM performance on clinical tasks like electrocardiogram (ECG) analysis is crucial.
• State-of-the-art LLMs from OpenAI and Google require rigorous assessment for ECG interpretation accuracy.

Purpose of the Study:

• To compare ChatGPT (GPT-5.2 Thinking) and Gemini (Gemini 3 Pro) in interpreting ECGs.
• To assess the accuracy and consistency of LLMs in identifying electrical axis and heart rhythm.
• To evaluate the current clinical usability of LLMs for cardiology and identify areas for improvement.

Main Methods:

• ECGs were sourced from the Lobachevsky University Electrocardiography Databases on PhysioNet.
• LLM responses were evaluated for first-shot accuracy and consistency across three trials.
• First-shot accuracy was analyzed by rhythm and axis type to identify systematic trends.

Main Results:

• Both ChatGPT and Gemini exhibited comparable first-shot accuracies for electrical axis and rhythm classification.
• Both models encountered difficulties with less common rhythms such as multifocal rhythms and tachycardias.
• Macro F1 analysis revealed low overall classification performance for both LLMs in axis and rhythm interpretation, with ChatGPT outperforming Gemini slightly on axis classification.

Conclusions:

• Macro F1 scores indicate that neither ChatGPT nor Gemini are reliable for independent clinical diagnoses in cardiology.
• Both LLMs demonstrated challenges in interpreting ECGs for rhythm and axis identification.
• This study aims to inform the continued development of LLMs for enhanced physician assistance in cardiology.