Taking a Deep Look at Multi-rater Agreement for Calibrated Medical Image Segmentation

In medical image analysis, it is common to collect multiple annotations from different clinical experts or raters to mitigate potential diagnostic inaccuracies. However, in computer vision research, it has been conventional to adopt ground-truth labels generated either by majority voting or by selecting a single rater's annotation. Such practices overlook the rich information embedded in the original multi-rater annotations, particularly the patterns of agreement and disagreement among experts. In this paper, we propose to explicitly model multi-rater (dis-)agreement, referred to as MRNet+, which offers several key innovations. First, an Expertise-aware Inferring Module (EIM) is developed to incorporate the expertise level of each rater as prior knowledge, thereby forming discriminative semantic features. Second, our method efficiently infers interobserver variability through a disentangled erasing-based Multi-rater Knowledge Modeling (MKM) module. Third, the cues pertaining to multi-rater (dis-)agreement are further leveraged through a Multi-rater Perception Module (MPM) to enhance segmentation performance. To the best of our knowledge, this is the first method to generate calibrated predictions that account for varying expertise levels in medical image segmentation. Moreover, MRNet+ is computationally efficient, achieving real-time performance with a speed of 89 FPS on 256×256 images. We validate our framework across ten medical segmentation tasks spanning diverse imaging modalities. In all cases, MRNet+ consistently outperforms state-of-the-art specialized methods, highlighting its effectiveness and broad applicability to real-world medical image segmentation scenarios. Source code is available at https://github.com/jiwei0921/MRNet_plus.