Instruction-Tuned Large Language Models for Clinical Data Extraction: Creating an Aortic Measurement Database from CT...

Chest computed tomography (CT) is commonly used to evaluate thoracic aortic dilations and aneurysms, but aortic measurements are often embedded in unstructured radiology report narratives, limiting their accessibility for clinical decision-making and research. Structured extraction of these measurements could flag incidental findings in the electronic health record, improve adherence to surveillance and surgical referral guidelines, and support population-level studies on aortic disease. We compared the performance of fine-tuned BERT-based models with instruction-tuned, few-shot, and zero-shot Llama large language models for extracting aortic diameters from 356,690 institutional chest CT reports spanning 2013 to 2023. A subset of 2010 reports was manually annotated, with 1002 used for training, 504 for validation, and 504 for testing. Aortic diameters were annotated at eight standardized anatomical sites. The instruction-tuned Llama 3.1 model achieved the highest performance, with a macro F1 score of 0.992 on the validation set and 0.970 on the test set, outperforming the few-shot (F1 = 0.838) and zero-shot (F1 = 0.663) Llama 3.1 models, as well as the fine-tuned Clinical BERT model (F1 = 0.954). Applied to the full dataset, the instruction-tuned model identified aortic measurements in 49,387 of 356,690 reports (13.85%). These results demonstrate that instruction-tuned large language models enable high-fidelity extraction of clinically relevant information with minimal annotation, and the framework developed here is adaptable to a broad range of medical information extraction tasks from unstructured text.