Zero-shot Thoracic Oncologic History Generation for Radiologists Using Retrieval-augmented Large Language Model...

Area of Science:

• Artificial Intelligence in Medicine
• Oncology Informatics
• Natural Language Processing

Background:

• Gathering oncologic history is crucial but inefficient in clinical practice.
• Current large language model (LLM) summarization methods require manual effort and subjective evaluation, hindering clinical application.
• Developing automated, objective methods for generating clinical summaries is essential.

Purpose of the Study:

• To develop and evaluate a zero-shot LLM pipeline for programmatic generation of structured oncologic histories.
• To utilize retrieval-augmented generation to ground LLM outputs for improved accuracy and reliability.
• To assess the performance of different LLMs in summarizing oncologic history data.

Main Methods:

• Retrospective electronic health record data from thoracic oncology patients were analyzed.
• A retrieval-augmented generation pipeline was implemented to filter clinical data for summarization.
• Three HIPAA-compliant LLMs (GPT-4o mini, o3-mini, GPT-5-mini) generated structured summaries using a zero-shot prompt approach.

Main Results:

• The GPT-5-mini pipeline achieved the highest mean completeness score (95.5%) and accuracy (97.9% with o3-mini).
• GPT-4o mini demonstrated the fastest processing time per summary (12.7 seconds), significantly below the 240-second manual benchmark.
• Projected time savings could lead to substantial annual revenue increases per radiologist with minimal associated costs.

Conclusions:

• The developed LLM pipeline accurately summarizes oncologic history without manual fine-tuning or information retrieval.
• The completeness and speed of automated summaries vary depending on the specific LLM employed.
• This automated approach offers a promising solution for efficient and accurate clinical summarization in oncology.