Instruction-Tuned Large Language Models for Clinical Data Extraction: Creating an Aortic Measurement Database from CT...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine
• Radiology Informatics

Background:

• Chest computed tomography (CT) reports contain vital aortic measurements, but they are often unstructured.
• Accessibility of these measurements is limited, hindering clinical decision-making and research on aortic diseases.
• Structured data extraction can improve patient care and facilitate population studies.

Purpose of the Study:

• To compare the performance of large language models (LLMs) against traditional models for extracting aortic diameters from chest CT reports.
• To evaluate instruction-tuned, few-shot, and zero-shot Llama models against a fine-tuned BERT model.
• To assess the feasibility of using LLMs for high-fidelity medical information extraction.

Main Methods:

• A dataset of 356,690 chest CT reports (2013-2023) was used.
• A subset of 2010 reports was manually annotated for aortic diameters at eight anatomical sites.
• Instruction-tuned, few-shot, and zero-shot Llama 3.1 models were compared with a fine-tuned Clinical BERT model.

Main Results:

• The instruction-tuned Llama 3.1 model achieved the highest performance with a test set F1 score of 0.970.
• This model outperformed few-shot (F1=0.838), zero-shot (F1=0.663) Llama models, and Clinical BERT (F1=0.954).
• The instruction-tuned model identified aortic measurements in 13.85% of the full dataset.

Conclusions:

• Instruction-tuned LLMs enable high-fidelity extraction of aortic measurements from unstructured radiology reports with minimal annotation.
• This approach significantly improves accessibility of critical data for clinical decision-making and research.
• The developed framework is adaptable for various medical information extraction tasks.