EchoLLM: extracting echocardiogram entities with light-weight, open-source large language models.

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Data Extraction

Background:

• Large language models (LLMs) offer advantages in clinical information extraction over traditional methods.
• Proprietary LLMs present challenges related to privacy and cost, limiting accessibility.
• Open-source LLMs present a viable alternative for clinical applications.

Purpose of the Study:

• To evaluate 14 open-source large language models (LLMs) for extracting clinically relevant findings from echocardiogram reports.
• To assess the feasibility of implementing open-source LLMs in clinical information extraction workflows.
• To compare the performance of different open-source LLMs on this specific task.

Main Methods:

• 14 open-source LLM models were utilized to extract entities from 507 echocardiogram reports.
• Reports were manually annotated by healthcare professionals and adjudicated.
• Performance was measured using precision, recall, and F1 scores for 9 extracted entities.

Main Results:

• Gemma2:9b-instruct demonstrated the highest performance with precision, recall, and F1 scores of 0.973, 0.959, and 0.965, respectively.
• Phi3:3.8b-mini-instruct had the lowest precision (0.831).
• Gemma:7b-instruct exhibited the lowest recall (0.382) and F1 scores (0.392).

Conclusions:

• Open-source LLMs are feasible for extracting entities from echocardiogram reports.
• This approach can support clinical research and healthcare delivery.
• Utilizing open-source models facilitates more efficient computation and data extraction.