An Overlapping Multi-Signal Detection Framework Based on Masked Pretraining and Repulsion Loss

As the electromagnetic environment grows increasingly crowded with coexisting wireless systems, overlapping signals have become commonplace. Signal overlap in a limited space leads to feature mixing, boundary blurring, and annotation difficulty, which existing deep learning methods cannot effectively handle. To tackle these challenges, we propose a two-stage detection framework that combines SimMIM self-supervised pretraining with a repulsion loss. In the pretraining phase, a Swin-Tiny backbone is trained on unlabeled time-frequency images through masked reconstruction. In the fine-tuning phase, the backbone is transferred to a YOLOv8 detector, where the repulsion loss is optimized jointly with the base detection loss. In the component ablation, the complete framework improves mAP50 by 9.0 points over the random-initialized baseline on the validation set. On the held-out main and overlap-stress sets, it obtains 0.875 mAP50, 0.798 mAP50-95, and 0.873 mAR, and 0.838, 0.756, and 0.829, respectively. Compared with the YOLOv8 baseline, the observed mAP50 gains are 1.95 and 2.18 points, respectively. The main high-overlap subset is small; its observed 1.64-point margin is therefore reported as descriptive evidence. These results indicate that in-domain self-supervised pretraining combined with spatially constrained box regression provides an effective representation and detection strategy for overlapping signals.