Using large language models to automate summarization of CT simulation orders in radiation oncology

Area of Science:

• Radiation Oncology
• Artificial Intelligence
• Medical Informatics

Background:

• Radiation oncology therapists manually summarize Computed Tomography (CT) simulation orders.
• Manual summarization increases workload, introduces variability, and is prone to human errors.

Purpose of the Study:

• To address challenges in manual CT simulation order summarization.
• To automate summary generation using a large language model (LLM).
• To evaluate the performance of an LLM for this task.

Main Methods:

• Collected 607 CT simulation orders from the Aria database.
• Utilized a LLaMa 3.1 405B model via API for keyword extraction and summary generation.
• Developed customized prompts collaboratively with therapists and validated summaries against ground truth.

Main Results:

• LLM-generated summaries achieved over 98% accuracy compared to ground truth.
• Demonstrated improved consistency in format and enhanced readability.
• Showed consistent performance across diverse treatment modalities and disease sites.

Conclusions:

• LLMs, specifically LLaMa 3.1 405B, precisely and consistently summarize CT simulation orders.
• LLMs show significant potential to assist therapists, reduce workload, and enhance radiation oncology workflow efficiency.