Deep learning and machine learning integration of radiomics and transcriptomics predicts response-adapted...

Radiotherapy (RT) remains a cornerstone treatment for head and neck cancer squamous cell carcinoma. However, therapeutic responses vary considerably among patients due to radiation resistance, which limits long-term survival and contributes to recurrence and disease progression. Developing robust deep learning (DL) and machine learning (ML)-based predictive models is essential to improve response prediction, evaluate treatment outcomes, and identify biomarkers linked to radiosensitization.

This single-center retrospective study applied DL and ML models to analyze CT scans and RNA-seq gene expression data for prognostic and biomarker discovery purposes. For image analyses, two independent datasets were used. Dataset A includes 1,100 CT scans (pre- and post-treatment) from 476 patients with stage III and IV laryngeal carcinoma treated with response-adapted RT. A convolutional neural network (CNNs) integrated with a recurrent network (RNNs) was used for single-point tumor localization and response prediction. Dataset B, comprising 500 scans from 169 patients treated with radical RT, served as the additional validation cohort. Pre- and post-treatment scans were used to train a DL model, which showed better prediction performance for survival and disease-specific outcomes, including progression and locoregional recurrence. For gene expression-based biomarker analysis, TCGA data (n = 231) were examined using glmBoost, support vector machine classifier (SVM), and random forest (RF) algorithms to construct and predict genes associated with radiosensitivity, and the GSE20020 dataset was used to validate the model performance. Proteins and mRNA were used to confirm the signature biomarkers using qRT-PCR and LC-MS mass spectrometry.

For CT scan image analysis, the DL-model achieved AUCs of 0.792 (p = 0.031) at 2-month and 0.832 (p < 0.01) at 6-month follow-up. Risk scores significantly correlated with overall survival (HR 1.59, 95% CI 1.34-3.22, p = 0.063), progression-free survival (1.39, 95% CI 1.16-2.29, p = 0.103). The pathological response in dataset B was likewise significantly predicted by the model. Among 39 differentially expressed genes, ML-model analysis identified 13 candidate genes associated with radiosensitivity on repeated cross-validation with an AUROC of 0.91 in the training set. In the validation dataset, when the models were optimized, the models consistently predicted seven core genes, achieving AUCs ranging from 0.96 to 0.94 to predict the radiosensitivity.

These findings highlight the effectiveness of DL and ML approaches in integrating imaging and transcriptomic data to predict response-adapted RT response and patient outcomes. These automated, and interpretable AI-driven biomarkers hold significant potential for clinical translation. Future research should aim to expand datasets and validate the models in multicenter cohorts for broader applicability.