Deep learning analysis of MRI to assess rectal cancer treatment

Traditional neoadjuvant therapy for locally advanced rectal cancer (LARC) results in pathologic complete response (pCR) in approximately 15% of patients, supporting non-operative strategies for those with clinical complete response (cCR). The subjectivity and variability in MRI-based cCR assessments highlight the need for objective, quantitative tools.

To develop deep learning models for automated rectal tumor segmentation on pre- and post-treatment MRIs, and to identify radiomic features differentiating cCR from non-cCR patients.

We retrospectively analyzed pre- and post-treatment MRIs from 37 LARC patients enrolled in a Phase 2 TNT trial (NCT04380337). Rectal tumors were segmented on T2-weighted images by two data scientists, refined by a radiologist (reference standard), and independently segmented by a fellow. For pre-treatment segmentation, Model 1 (baseline; $n=37$ ) was trained on reference cases, then used to generate pseudo-labels for 81 additional cases. Model 2 (semi-supervised; $n=118$ ) was trained on the combined dataset. Model 3 (baseline; $n=37$ ) was trained on post-treatment cases. Radiomic features were extracted from post-treatment ADC maps, filtered by reproducibility (ICC $\ge 0.8$ ) and redundancy (Spearman $\rho \le 0.95$ ), then analyzed using unsupervised hierarchical clustering.

For pre-treatment segmentation, radiologist-fellow inter-rater agreement was DSC $=0.748\pm 0.092$ . Model 1 achieved mean DSC $=0.682\pm 0.254$ versus the radiologist, significantly lower than inter-rater agreement. Model 2 improved performance to mean DSC $=0.769\pm 0.214$ (mean gain $=0.087$ ; $12.8\%$ relative improvement; $p<0.001$ ), slightly outperforming inter-rater agreement. For post-treatment segmentation, inter-rater agreement declined to mean DSC $=0.362\pm 0.256$ , while Model 3 achieved mean DSC $=0.175\pm 0.231$ versus the radiologist, reflecting challenges from treatment-induced tissue changes affecting both automated models and human raters. Radiomic clustering revealed two distinct patient groups aligned with cCR and non-cCR status.

This study demonstrates the feasibility of deep learning-based automated segmentation and radiomic profiling for differentiating treatment response in rectal cancer. Semi-supervised learning with pseudo-labeled data significantly improved segmentation performance, offering a practical approach to overcome limited annotations. Radiomic features warrant validation in larger multi-center studies for clinical translation.