Improving Colorectal Cancer Screening and Risk Assessment through Predictive Modeling on Medical Images and Records

Colonoscopy screening effectively identifies and removes polyps before they progress to colorectal cancer (CRC), but current follow-up guidelines rely primarily on histopathologic features, overlooking other important CRC risk factors. Variability in polyp characterization among pathologists also hinders consistent surveillance decisions. Advances in digital pathology and deep learning enable the integration of pathology slides and medical records for more accurate progression risk prediction. Using data from the New Hampshire Colonoscopy Registry, including longitudinal follow-up, a transformer-based model for histopathology image analysis was adapted to predict 5-year progression risk. Multi-modal fusion strategies were further explored to combine clinical records with deep learning-derived image features. Training the model to predict intermediate clinical variables improved 5-year progression risk prediction [area under the receiver-operating characteristic curve (AUC), 0.630] compared with direct prediction (AUC, 0.615; P = 0.013). Integrating whole-slide imaging-based model predictions with nonimaging features further improved performance (AUC, 0.672), significantly outperforming the nonimaging-only approach (AUC, 0.666; P = 0.002). These results highlight the value of integrating diverse data modalities with computational methods to enhance progression risk stratification.