Cross-Institutional Validation of a novel LLM-Based Cardiac Event Extraction framework from Electronic Health Records

Area of Science:

• Cardio-oncology research
• Artificial intelligence in medicine
• Electronic Health Record (EHR) data analysis

Background:

• Cardiac toxicity is a major concern for cancer patients undergoing radiotherapy.
• Evaluating cardiac events is complex, time-consuming, and resource-intensive, hindering cardio-oncology research progress.
• Large-language models (LLMs) offer potential for automated cardiac event identification.

Purpose of the Study:

• To develop and validate a novel LLM-based automated cardiac event identification framework.
• To assess the framework's accuracy and efficiency using a two-institution dataset.
• To enable scalable cardio-oncology research by reducing resource requirements for cardiac event identification.

Main Methods:

• Analyzed data from 411 lung and breast cancer patients across two institutions.
• Developed the Two-phase Reasoning for Automated Cardiac Event Recognition (TRACER) framework, combining structured data and LLM analysis of unstructured clinical notes.
• Validated three top-performing open-source LLMs (DeepSeek-R1, Llama-3.3, Mistral-Large) against physician-adjudicated ground truth.

Main Results:

• The TRACER framework achieved high accuracy across development (79.4%), internal validation (81.0%), and external validation (79.3%) cohorts.
• DeepSeek-R1 and Llama-3.3 demonstrated strong performance, with Llama-3.3 reaching 85.5% accuracy on external validation.
• TRACER processing time was significantly reduced (20-42 seconds/patient) compared to manual review (2 hours/patient).

Conclusions:

• The locally deployed TRACER LLM framework accurately identified cardiac events across institutions and cancer types.
• This automated approach substantially reduces the resources needed for cardiac event identification.
• TRACER facilitates scalable cardio-oncology research by streamlining data analysis.