A twin-tower model using MRI and gene for prediction on brain tumor patients' response to therapy

Glioma is the most prevalent and aggressive primary brain tumor, with a poor prognosis of patients and a high mortality rate. Standard treatment of surgery, radiation, and chemotherapy may not be effective for some patients as they suffer from a stable progression of disease after treatment. Hence, it is crucial to predict the patient's response to therapy as a guide for the treatment plan. In this paper, we propose a multimodal model based on both magnetic resonance imaging and genomic data. As the dataset has a majority of single-modality samples with a few ratios of multi-modality samples, we propose a twin-tower architecture to solve the unimodal dominance issue and fully use the single-modality data.

The proposed architecture comprises an image encoder and a gene encoder trained on the single-modality samples for feature extraction, along with a classification head trained on multi-modality samples. In this way, all the single-modality samples can be beneficial to the whole model, and the need for the multi-modality is diminished. The proposed model outperforms the comparison methods across all metrics, achieving an accuracy of 85% on the cross-validation. The ablation experiment comparing the proposed architecture with single-modality models reflects the effectiveness of the proposed twin-tower architecture.

The proposed model exhibits excellent scalability and can accommodate the integration of additional modalities without the requirement of too many multi-modality samples.