Related Experiment Video
Updated: Aug 5, 2026
Foreign Accent and Forensic Speaker Identification in Voice Lineups: The Influence of Acoustic Features Based on Prosody
Published on: September 27, 2024
Case-matched retrieval improves textual alignment of LLM-generated radiology impressions
Vera Sorin1, Jeremy D Collins1, Lewis D Hahn1
1Department of Radiology, Mayo Clinic College of Medicine and Science, Mayo Clinic, Rochester, Minnesota, United States of America.
Background:
Radiology impressions guide clinical care. Large Language Models (LLMs)-drafted impressions can drift into generic, off-style text. Retrieval-augmented generation (RAG) enables context-aware few-shot prompting during inference.
Methods:
This retrospective IRB-approved study included 11,998 CT pulmonary angiography (CTPA) reports. We built a retrieval bank from 11,399 reports and reserved 599 reports for testing. GPT-4o and LLaMA 3.1-70B generated impressions from the "findings" section using three setups: zero-shot, fixed random few-shot, and dynamic retrieval-selected few-shot (top-k semantic matches; k = 3/5/10). We ran temperatures 0, 0.7, 1. We scored outputs against the original impressions with ROUGE and BERTScore F1, report mean scores with 95% confidence intervals, and tested for statistical significance using Wilcoxon signed-rank test.
Results:
Dynamic retrieval-based few-shot prompting outperformed zero-shot and fixed few-shot prompting across all configurations (all p < 0.05). The highest scores were observed at temperature 0 and k = 10. ROUGE-1 F1 increased to 0.44-0.47 for GPT-4o and 0.37-0.50 for LLaMA, versus 0.35-0.37 and 0.25-0.37, respectively, in zero-shot prompting. Lower temperature and larger k were associated with higher similarity scores.
Conclusions:
Dynamic, case-matched retrieval improved alignment of LLM-generated CTPA impressions with reference impressions on automated text-similarity metrics. Scores remained moderate, and radiologists' verification is still required before clinical deployment.
