Related Experiment Video
Updated: Apr 15, 2026
03:14
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
1.3K
Leveraging Large Language Models for Automated Extraction of Abdominal Aortic Aneurysm Features from Radiology
Praneel Mukherjee1, Ryan C Lee1,2, Roham Hadidchi1
1Department of Radiology, Montefiore Medical Center and Albert Einstein College of Medicine, Bronx, NY 10461, USA.
Diagnostics (Basel, Switzerland)
|April 14, 2026
Summary
Large language models (LLMs) show promise in automatically extracting abdominal aortic aneurysm (AAA) data from CT reports. Larger, medically trained LLMs performed better, but careful validation is needed for clinical use.
Area of Science:
- Radiology and Medical Informatics
- Artificial Intelligence in Healthcare
- Natural Language Processing for Clinical Data
Background:
- Radiology reports for abdominal aortic aneurysms (AAA) contain vital clinical data.
- Manual extraction of AAA information from CT reports is time-consuming and inefficient.
- Automated extraction using large language models (LLMs) could improve efficiency.
Purpose of the Study:
- To evaluate the performance of open-source LLMs in extracting key AAA findings from abdominal CT radiology reports.
- To compare the accuracy and agreement of different LLMs for specific AAA-related data points.
- To analyze LLM reasoning for accurate and erroneous extractions.
Main Methods:
- Retrospective analysis of 500 abdominal CT reports mentioning AAA.
- Manual review to establish ground truth labels for aneurysm presence, size, rupture, and prior repair.
- Evaluation of four open-source LLMs (Qwen2.5-7B-Instruct, Llama3-Med42-8B, GPT-OSS-20B, MedGemma-27B-text-it) for data extraction.
- Comparison of LLM outputs against ground truth using exact-match accuracy and Fleiss' kappa.
- Examination of reasoning traces to understand model behavior.
Main Results:
- High accuracy (0.90-0.95) for AAA presence and prior repair extraction.
- Variable accuracy for aneurysm size (0.67-0.88) due to class imbalance and dimension misselection.
- Accurate identification of rupture status (>0.90) but lower inter-model agreement.
- Larger models (GPT-OSS-20B, MedGemma-27B) generally outperformed smaller ones.
- Reasoning analysis revealed strengths in measurement but errors in dimension selection and over-inference.
Conclusions:
- LLMs can accurately extract critical AAA information from radiology reports with interpretable reasoning.
- Medically trained and larger LLMs demonstrate superior performance.
- Task-specific performance varies, necessitating rigorous validation and human-in-the-loop systems for clinical deployment.
