Semi-supervised mp-MRI data synthesis with StitchLayer and auxiliary distance maximization

The availability of a large amount of annotated data is critical for many medical image analysis applications, in particular for those relying on deep learning methods which are known to be data-hungry. However, annotated medical data, especially multimodal data, is often scarce and costly to obtain. In this paper, we address the problem of synthesizing multi-parameter magnetic resonance imaging data (i.e. mp-MRI), which typically consists of Apparent Diffusion Coefficient (ADC) and T2-weighted (T2w) images, containing clinically significant (CS) prostate cancer (PCa) via semi-supervised learning and adversarial learning. Specifically, our synthesizer generates mp-MRI data in a sequential manner: first utilizing a decoder to generate an ADC map from a 128-d latent vector, followed by translating the ADC to the T2w image via U-Net. The synthesizer is trained in a semi-supervised manner. In the supervised training process, a limited amount of paired ADC-T2w images and the corresponding ADC encodings are provided and the synthesizer learns the paired relationship by explicitly minimizing the reconstruction losses between synthetic and real images. To avoid overfitting limited ADC encodings, an unlimited amount of random latent vectors and unpaired ADC-T2w Images are utilized in the unsupervised training process for learning the marginal image distributions of real images. To improve the robustness for training the synthesizer, we decompose the difficult task of generating full-size images into several simpler tasks which generate sub-images only. A StitchLayer is then employed to seamlessly fuse sub-images together in an interlaced manner into a full-size image. In addition, to enforce the synthetic images to indeed contain distinguishable CS PCa lesions, we propose to also maximize an auxiliary distance of Jensen-Shannon divergence (JSD) between CS and nonCS images. Experimental results show that our method can effectively synthesize a large variety of mp-MRI images which contain meaningful CS PCa lesions, display a good visual quality and have the correct paired relationship between the two modalities of a pair. Compared to the state-of-the-art methods based on adversarial learning (Liu and Tuzel, 2016; Costa et al., 2017), our method achieves a significant improvement in terms of both visual quality and several popular quantitative evaluation metrics.