Zero-shot Thoracic Oncologic History Generation for Radiologists Using Retrieval-augmented Large Language Model...

Background Gathering oncologic history is necessary but inefficient. Recent efforts to generate clinical summaries using large language models (LLMs) have relied on manual information retrieval, prompt tuning, and subjective evaluation, limiting clinical translation. Purpose To develop and evaluate a zero-shot LLM pipeline that programmatically generates structured oncologic histories using retrieval-augmented generation to ground LLM outputs. Materials and Methods Retrospective electronic health record data were collected for thoracic oncology patients at a cancer center from January 2018 to April 2024. Twenty surveyed radiologists selected 10 clinical parameters for summarization. A retrieval-augmented generation pipeline was developed to filter clinical data. Three Health Insurance Portability and Accountability Act-compliant LLMs-OpenAI's GPT-4o mini, o3-mini, and GPT-5-mini-produced structured summaries using a defined data schema and a zero-shot prompt approach. Summaries were manually evaluated against retrieved clinical text, with completeness reference standard based on the output of GPT-5-mini. Two radiology residents were timed summarizing clinical parameters, establishing a 240-second benchmark. Comparisons between models were conducted using paired t tests for accuracy, completeness, cost, and processing time. Cost-benefit analyses were conducted using manual extraction time, automated summary generation time, and national survey data on radiologist productivity. Results Of 2433 patients, 50 were randomly selected for LLM summarization (mean age, 67.0 years ± 12.5 [SD]; 32 female patients). The GPT-5-mini pipeline achieved the highest mean completeness score of 95.5% (95% CI: 94.2, 96.7) and was similar to o3-mini for a mean accuracy of 97.9% (95% CI: 96.8, 99.1). GPT-4o mini had the fastest processing time per summary, averaging 12.7 seconds compared with the 240-second benchmark (P < .001). Ideally, these time savings project to annual incremental revenue of $19 525 to $70 942 per radiologist, with costs of $30.26 to $32.46. Conclusion The LLM pipeline, which does not require manual fine-tuning or information retrieval, accurately summarizes oncologic history. The completeness and speed of these summaries vary depending on the LLM used. © RSNA, 2026 Supplemental material is available for this article. See also the editorial by Mohajer and Terwiesch in this issue.