Siamese Delta Network with Multimodality Fusion for Systemic Treatment Response Prediction in Pancreatic Cancer

Systemic treatment is a main way for pancreas cancer patients that are ineligible for surgery. A subgroup of patients showed good response to systemic treatment and the rest received limited benefits. CT images provide a non-invasive way to assess the treatment response. Alternative non-image methods include radiology analysis, tumor marker analysis and combination analysis. To combine the image and non-image data, we propose the Siamese Delta Network with Multimodality Fusion (SDN-MF) to predict systemic treatment response in an end-to-end way. First, a Siamese Delta Network (SDN) is designed to process pre-treatment and pre-surgery CT images and get the image feature changes to predict response. Then, patients' characteristics from EMR and alternative analysis results forms non-image data, which is incorporated into SDN with a multimodality fusion (MF) module. The proposed SDN-MF is evaluated on a private dataset and achieves average AUC value of 0.883 with five cross-validation. Comparison among image-only, non-image-only, and fusion models verifies the superior of multimodality model in predicting systemic treatment response of pancreas cancer patients.