Spatio-Temporal Convolutional LSTMs for Tumor Growth Prediction by Learning 4D Longitudinal Patient Data

Area of Science:

• Artificial Intelligence in Medical Imaging
• Deep Learning for Predictive Modeling
• Computational Oncology

Background:

• Prognostic tumor growth modeling using volumetric medical imaging is crucial for treatment planning and surgical interventions.
• Deep learning, particularly convolutional networks (ConvNets), shows promise in predicting tumor volumes, surpassing traditional mathematical models.
• Existing 2D ConvNet approaches fail to fully utilize the spatio-temporal context of 4D longitudinal patient data and are limited to volume prediction.

Purpose of the Study:

• To develop a deep learning model that leverages the full spatio-temporal context of 4D longitudinal tumor data for more accurate prognostic modeling.
• To extend predictive capabilities beyond tumor volume to include other clinically relevant properties like cell density and CT intensity.
• To validate the model's performance against existing methods using a large dataset and demonstrate its generalizability in medical image segmentation.

Main Methods:

• Formulation of tumor growth using a novel Spatio-Temporal Convolutional Long Short-Term Memory (ST-ConvLSTM) network.
• The ST-ConvLSTM model jointly learns inter-slice 3D contexts and longitudinal temporal dynamics from multiple patient studies.
• Integration of non-imaging patient data and end-to-end training capabilities were incorporated into the model architecture.

Main Results:

• The ST-ConvLSTM model achieved a Dice score of 83.2%±5.1% and a Relative Volume Difference (RVD) of 11.2%±10.8% in predicting future tumor volumes.
• These results were statistically significantly superior (p < 0.05) to traditional linear models, ConvLSTM, and Generative Adversarial Networks (GANs).
• The model successfully predicted cell density and CT intensity, and demonstrated generalizability by achieving an 86.3%±1.2% Dice score for left-ventricle segmentation in 4D ultrasound.

Conclusions:

• The proposed ST-ConvLSTM model effectively utilizes spatio-temporal information from 4D medical imaging for accurate tumor growth prediction.
• This deep learning approach offers enhanced predictive power beyond tumor volume, including tissue characteristics, and shows significant clinical potential.
• The ST-ConvLSTM model's demonstrated generalizability suggests its broad applicability in various 4D medical image analysis and segmentation tasks.