Information extraction from clinical notes: are we ready to switch to large language models?

Area of Science:

• Natural Language Processing
• Artificial Intelligence in Medicine
• Clinical Informatics

Background:

• Clinical information extraction (IE) is crucial for leveraging unstructured clinical notes.
• Large Language Models (LLMs) like LLaMA-2 and LLaMA-3 are emerging as powerful tools for NLP tasks.
• Comparing LLMs with established models like BERT is essential for understanding their utility in healthcare.

Purpose of the Study:

• To evaluate the performance, generalizability, and computational efficiency of instruction-tuned LLaMA-2 and LLaMA-3 models.
• To benchmark LLaMA models against BERT for clinical Named Entity Recognition (NER) and Relation Extraction (RE).
• To assess the impact of data availability on model performance.

Main Methods:

• Development of a comprehensive annotated corpus of 1588 clinical notes from four diverse data sources.
• Instruction-tuning of LLaMA-2 and LLaMA-3 models for clinical NER and RE tasks.
• Benchmarking LLaMA models against BERT using metrics like F1 scores across different datasets.

Main Results:

• LLaMA models consistently outperformed BERT across all evaluated datasets.
• Significant performance gains were observed with LLaMA-3-70B, especially in low-data settings and on unseen data (over 7% improvement in NER, 4% in RE).
• LLaMA models exhibited increased computational costs, requiring more memory and GPU hours, and running up to 28 times slower than BERT.

Conclusions:

• Instruction-tuned LLaMA models demonstrate strong potential for clinical NER and RE.
• A critical tradeoff exists between enhanced performance and increased computational demands of LLaMA models.
• Careful consideration of resource constraints and application-specific needs is vital when selecting between LLMs and BERT for clinical IE.