Weakly supervised commissioning of externally developed auto-segmentation models and applied to male pelvis MR...

When introducing an auto-segmentation model into clinical practice, assessing the quality of the predicted segmentations and the robustness of the model over a wide range of anatomical variation and/or image quality is difficult. Especially, when the model is provided by an external party and the institution introducing the model does not possess a high-quality dataset to commission the model on.

Assuming that a model is more likely to fail for an atypical case as opposed to a more average one, we propose a methodology that selects cases for commissioning using unsupervised anomaly detection. For this, the model supplier provides a set of image/shape features that correlate with model performance on the training data. Next, the receiving hospital can use these features to train an unsupervised anomaly detector on a large dataset of unlabeled cases and use the anomaly scores to select representative cases for commissioning of the model. Since the anomaly detector is trained on unlabeled data, a large, high-quality, curated dataset is not required on the receiving hospital side.

Using the proposed approach, the likelihood of selecting atypical edge cases with low segmentation performance was increased, as compared to random selection. For a selection of 20 cases, an increase of 22% was observed.

The increased performance spread provides a more representative range of expected performance in clinical practice. This approach could be used for model commissioning to increase the confidence that the model performs well over a wide range of expected anatomical variation.