Knowledge relay: Synergetic generation and transfer learning for pancreatic tumor segmentation on multimodal images

Pancreatic cancer is among the most lethal malignancies, with the lowest survival rates. The use of image-guided radiotherapy has shown significant potential in enhancing surgical outcomes for pancreatic cancer. However, accurate segmentation of pancreatic tumors prior to radiotherapy remains a challenge due to the small size, irregular shape, and indistinct boundaries of the pancreas and tumor in monomodal imaging. Furthermore, the availability of multimodal images that meet the requirements for precise pancreatic segmentation is highly limited, leading to the datasets that fail to provide comprehensive knowledge for effective image representation in pancreatic tumor segmentation.

This study aims to develop a method for accurately segmenting pancreatic tumors under very harsh data conditions, in which the currently available datasets are fragmented with the issues, such as limited sample sizes, inconsistent lesion matching, and incomplete modalities.

We propose a knowledge relay framework that leverages synergistic generation and transfer learning strategies. The relay comprises three batons: pancreatic PET image generation, coarse detection, and fine segmentation. Multimodal images, including CT, MR, and PET from three separate datasets, are integrated within this framework. The knowledge contained in each dataset is sequentially transferred and aggregated through the batons by the strategies of transfer learning and fine-tuning. Additionally, we introduce a mask-constrained CycleGAN and an inter-attention UNet within this framework to enhance the extraction and utilization of knowledge for accurate pancreatic tumor segmentation.

The proposed knowledge relay framework achieves the state-of-the-art performance in pancreatic tumor segmentation on PET/MR images. On the images collected from 19 subjects, our method attained a $\text{DSC}$ of 80.06%, $\text{SEN}$ of 83.39%, $\text{SPE}$ of 99.81%, $\text{ASD}$ of 4.87 $\text{mm}$ , and $95\text{HD}$ of 12.69 $\text{mm}$ .

The results of comparison and ablation experiments validate the effectiveness of the proposed knowledge relay framework in extracting and integrating knowledge from fragmented datasets under constrained conditions. The comprehensive and enriched knowledge significantly enhances the accuracy of pancreatic tumor segmentation.