HybPromptSeg: Spatial-frequency hybrid prompting for unified multi-task medical image segmentation

Fully automated segmentation of medical images is essential for streamlining clinical workflows and improving diagnostic efficiency. However, building a single model that jointly handles multiple segmentation tasks remains challenging because different tasks may vary not only in target structures but also in heterogeneous data distributions. Existing learnable-prompt methods encode task-specific priors, but their generalization may still be constrained by cross-task interference when the tasks have substantially different data distributions. In this study, we propose Hybrid Prompt-based Segmentation, referred to as HybPromptSeg, which is a hybrid prompt-driven, encoder-decoder-based network that unifies multiple medical image segmentation tasks within a single shared model. Concretely, we introduce a Unified Hybrid Prompt (UHP) mechanism that integrates spatial-domain and Fourier-amplitude representations within a shared learnable prompt to model task- and modality-related variations. We further design a Task-conditioned Prompt Generation (TPG) module that generates discriminative task-conditioned prompts to guide the decoder for different segmentation tasks. To evaluate HybPromptSeg, we construct two benchmarks: a single-modality multi-task (SMMT) benchmark for evaluating task discrimination under shared-modality conditions, and a multi-modality multi-task (MMMT) benchmark for evaluating robustness under cross-modality variations. Experimental results demonstrate that HybPromptSeg achieves the best performance among the evaluated unified multi-task models on both benchmarks. The code is publicly available at https://github.com/nnnsjtu/HybSeg.