Cross-Domain Robust Pruning for Polyp Segmentation: Multi-Encoder Feature Fusion Beats Single-Encoder Baselines

Medical image segmentation requires dense pixel-level annotations, making large-scale dataset construction expensive and motivating research into data-efficient training. The main objective of this paper is to determine whether fusing two complementary pretrained image encoders into a single similarity space can make training-free dataset pruning robust across heterogeneous polyp segmentation domains and to quantify that robustness against a comprehensive panel of baselines. To achieve this, we propose Multi-Encoder Diverse Pruning (MEDP), a training-free dataset-pruning method. MEDP fuses features from an ImageNet-pretrained ResNet-18 and a self-supervised DINOv2 ViT-S/14 into a single 896-D similarity space. It partitions the training pool via Louvain modularity maximization and selects per-community samples via maximal-marginal-relevance (MMR) ranking, which effectively balances eigenvector centrality with feature-space diversity. We benchmarked MEDP against 12 baselines at a 20% retention ratio across three polyp segmentation settings (Kvasir-SEG, CVC-ClinicDB, and a Combined cross-domain pool) using a standard 5-level UNet. Based on approximately 330 controlled training runs, the results demonstrate that MEDP achieves the highest mean test Dice of 0.7324 on the most challenging Combined cross-domain pool, significantly outperforming uniform random sampling (Cohen's d = +1.79, paired Wilcoxon p = 0.002). Conversely, all hand-crafted structure-aware variants failed to outperform uniform random sampling. These findings confirm that combining multi-encoder features with MMR diversity provides a simple and effective strategy for improving robustness across heterogeneous medical imaging settings and that the choice of pretrained image encoder is the dominant factor in segmentation-aware pruning.