Large-language-model empowered 3D dose prediction for intensity-modulated radiotherapy.

Area of Science:

• Medical imaging
• Radiotherapy
• Artificial Intelligence

Background:

• Radiotherapy treatment planning is complex, time-consuming, and resource-intensive.
• Accurate dose-volume histogram (DVH) prediction is crucial for automating radiotherapy planning.
• Deep learning models show potential for predicting DVHs from medical images.

Purpose of the Study:

• To develop and evaluate a deep learning model for predicting DVHs from medical images.
• To integrate a large-language model (LLM) for enhanced DVH prediction and clinician interaction.
• To implement an online human-AI collaboration system for automated intensity-modulated radiotherapy (IMRT) planning.

Main Methods:

• A pipeline was developed to convert unstructured images into a structured graph with image-patch and dose nodes.
• A novel Dose Graph Neural Network (DoseGNN) model was created for DVH prediction from the structured graph.
• The DoseGNN model was enhanced with an LLM to incorporate clinical knowledge and interactive instructions.

Main Results:

• DoseGNN demonstrated superior accuracy in DVH prediction compared to Swin Transformer, 3D U-Net CNN, and MLP models.
• For planning target volume (PTV), DoseGNN achieved significantly lower mean absolute errors (MAEs) than baseline models.
• For organs-at-risk (OARs), DoseGNN also showed substantial MAE reductions compared to the best baseline models.
• The LLM integration enabled seamless adjustment of treatment plans through natural language interaction with clinicians.

Conclusions:

• DoseGNN is a novel deep learning model for predicting radiation doses from medical images.
• The LLM-enhanced DoseGNN facilitates clinician interaction for treatment plan adjustments.
• Preliminary results indicate DoseGNN's potential for accurate DVH prediction and streamlined automated treatment planning through online clinician-AI collaboration.