Related Experiment Video
Updated: May 5, 2026
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Large Language Models for Accurate Medical Chart Abstraction: Enabling Scalable and Secure Artificial Intelligence
Zhusi Zhong1,2, Carl M Porto1, David Hong1
1From the Warren Alpert Medical School of Brown University (Z.Z., C.M.P., D.H., M.G., R.K., G.K., L.B., J.R.F., E.S., S.C., L.S., G.B., M.J., S.Y., Z.J., D.N.W.), Providence, Rhode Island.
Background And Purpose:
Medical chart abstraction plays a critical role in clinical research and quality monitoring by transforming unstructured narratives in the procedure reports into structured variables for large-scale analysis. We developed and evaluated a prompting-based large language model (LLM) framework for automated extraction of structured clinical variables from neurointerventional procedure reports in patients with acute ischemic stroke due to large vessel occlusions.
Materials And Methods:
This retrospective study included 2416 free-text neurointerventional acute stroke intervention (thrombectomy) reports with key radiology findings from 3 hospitals. Eight clinically relevant variables were annotated by hospital staff (without formal clinical training) and used as the nonexpert reference standard. Twenty-two instruction-tuned open-source LLMs (LLaMA, Qwen, Gemma, etc) were evaluated across architectures, sizes, and biomedical adaptations using 2 prompting strategies: Quick Response and Chain-of-Thought (CoT). Model performance was benchmarked against nonexpert staff annotations and medical expert ratings. Extraction accuracy, latency, and agreement with expert adjudication were assessed.
Results:
LLaMA3.3-70B achieved the highest overall accuracy (94.8%). CoT prompting improved performance on inferential variables (eg, site of occlusion), while Quick Response was optimal for directly stated procedural fields (eg, stent placement). Expert adjudication confirmed that LLaMA3.3-70B outperformed nonexpert annotations in 7 of 8 variables and matched junior medical students. Annotation accuracy increased with clinical experience, and artificial intelligence predictions were more closely aligned with expert interpretations than with those of nonexpert staff, especially for structured variables like IV tPA, TICI Post, and NIHSS.
Conclusions:
Prompted LLMs can accurately and scalably extract critical clinical information from neurovascular radiology reports without custom preprocessing, supporting integration into retrospective research pipelines and automated stroke registry curation.
