Intrinsic dimension for evaluating and analyzing neural networks for medical image segmentation

Evaluating medical image segmentation models typically relies on expert annotations, which are scarce, costly, and time-consuming to obtain. Intrinsic dimension (ID), defined as the minimal number of parameters required to describe high-dimensional data, has been shown to correlate with model generalization in classification tasks, but whether this relationship extends to segmentation remains unclear.

In this study, we systematically investigate ID as a label-free indicator for evaluating segmentation performance, including the generalization of standard segmentation models and the adaptability of promptable foundation models (i.e., SAM-based models). We also perform a layer-wise ID analysis to characterize variations in latent representation complexity. Furthermore, we propose a novel, computation- and memory-efficient strategy for ID calculation to broaden its applications.

We reveal the following findings: 1) For standard segmentation, lower ID values of training-sample representations are consistently associated with better generalization performance on unseen test samples, as measured by DSC. 2) For promptable segmentation, test-data ID consistently reflects model adaptability to specific target data compared to entropy, a metric commonly believed to predict performance. 3) Layer-wise ID in segmentation follows an initially increasing and then decreasing pattern, similar to that in classification.

Our findings suggest the potential of ID as a label-free indicator for assessing the generalization and adaptability of segmentation models, thereby supporting model selection in label-scarce clinical settings.