Evaluating large language models in echocardiography reporting: opportunities and challenges.

Area of Science:

• Cardiology
• Artificial Intelligence
• Medical Informatics

Background:

• Diagnostic echocardiography is crucial but faces challenges in timely, high-quality report generation.
• Large Language Models (LLMs) show promise in clinical text summarization but are underutilized for echocardiography reports.

Purpose of the Study:

• To adapt and evaluate open-source LLMs for summarizing echocardiography reports from 'Findings' to 'Impressions.'
• To assess the performance of fine-tuned LLMs against cardiologist-generated reports using quantitative and qualitative measures.

Main Methods:

• Adult echocardiography reports from Mayo Clinic (2017) were used for model development and validation.
• Open-source LLMs (Llama-2, MedAlpaca, Zephyr, Flan-T5) were fine-tuned using In-Context Learning and Quantized Low-Rank Adaptation.
• Performance was evaluated using automatic metrics and qualitative review by cardiologists, comparing AI-generated summaries to cardiologist-generated 'Impressions.'

Main Results:

• EchoGPT, a fine-tuned Llama-2 model, demonstrated superior performance, achieving win rates of 87%–99% in automatic metrics.
• Qualitative review showed EchoGPT reports were comparable to cardiologist-generated summaries, significantly preferred for conciseness (P < 0.001).
• Automatic metrics showed fair to modest correlations with human evaluation, with some insensitivity to measurement number changes.

Conclusions:

• EchoGPT can generate draft echocardiography reports, aiding in workflow streamlining for human review and approval.
• Further development of scalable evaluation methods for AI-generated echocardiography reports is necessary.