ECG-LM: Understanding Electrocardiogram with a Large Language Model

Background: The electrocardiogram (ECG) is a valuable, noninvasive tool for monitoring heart-related conditions, providing critical insights. However, the interpretation of ECG data alongside patient information demands substantial medical expertise and resources. While deep learning methods help streamline this process, they often fall short in integrating patient data with ECG readings and do not provide the nuanced clinical suggestions and insights necessary for accurate diagnosis. Methods: Although recent advancements in multi-modal large language modeling have propelled their application scope beyond the natural language processing domain, their applicability to ECG processing remains largely unexplored, partly due to the lack of text-ECG data. To this end, we develop ECG-Language Model (ECG-LM), the first multi-modal large language model able to process natural language and understand ECG signals. The model employs a specialized ECG encoder that transforms raw ECG signals into a high-dimensional feature space, which is then aligned with the textual feature space derived from the large language model. To address the scarcity of text-ECG data, we generated text-ECG pairs by leveraging detailed ECG pattern descriptions from medical guidelines, creating a robust dataset for pre-training ECG-LM. Additionally, we fine-tune ECG-LM with public clinical conversation datasets and build an additional supervised fine-tuning dataset based on real clinical data from the hospital, aiming to provide a more comprehensive and customized user experience. Results: ECG-LM outperforms existing few-shot and zero-shot solutions in cardiovascular disease detection across all 3 tasks (diagnostic, rhythm, and form) while also demonstrating strong potential in ECG-related question answering. Conclusions: The results across various tasks demonstrate that ECG-LM effectively captures the intricate features of ECGs, showcasing its versatility in applications such as disease prediction and advanced question answering.