A Dual-Domain Model Merging based Hybrid-Supervised Learning Framework for Computer-Aided Diagnosis

Self-supervised learning (SSL) can effectively alleviate the issue of limited training samples for Computer-Aided Diagnosis (CAD) by pre-training backbone with unlabeled data. Although the Dual-domain SSL can effectively capture complementary information from both spatial and frequency domains, the selected backbone for downstream task generally does not contain enough information from another domain. Since model merging technique can efficiently integrate the parameters of multiple networks into a unified network, a Dual-Domain Model Merging based Hybrid-Supervised Learning (DM-HSL) framework is proposed to improve the diagnostic performance of a CAD model. It simultaneously performs a dual-domain Masked Image Modeling (MIM) task and a classification task, where the latter applies the merged encoder network from the spatial domain and frequency domain encoders as backbone for the downstream task. The model merging module adopts diagnosis labels as additional supervised information to dynamically update the parameters of the three networks, so that the merged backbone can well fuse dual-domain information. Moreover, a novel GG-Loss contrastive loss is further developed to implement contrastive learning by integrating Gramian Metric of Representation Alignment and Generalized Cauchy-Schwarz divergence into a loss item to achieve unified alignment and collaborative optimization of representations among spatial domain, frequency domain and hybrid domain. Thus, this hybrid supervised dual-domain pre-training effectively improves the performance of backbone for the following fine-tuning in downstream task. The experiments on two medical image datasets demonstrate the effectiveness of the proposed DM-HSL.