Revisiting model scaling with a U-net benchmark for 3D medical image segmentation

Are larger models always better for 3D medical image segmentation? Despite the widespread adoption of 3D U-Net in various medical imaging tasks, this critical question remains underexplored. To challenge the common assumption, we systematically benchmark 18 U-Net variants-adjusting resolution stages, depth, and width-across 42 diverse public datasets. Our findings reveal that the answer is no: optimal architectures are highly task-specific, with smaller models often performing competitively. Specifically, we identify three key insights: (1) increasing resolution stages provides limited benefits for datasets with larger voxel spacing; (2) deeper networks offer limited advantages for anatomically complex shapes; and (3) wider networks provide minimal advantages for tasks with limited segmentation classes. Based on these insights, we provide practical guidelines for optimizing U-Net architectures according to dataset characteristics. Our findings highlight the limitations of the"bigger is better"paradigm while establishing a framework for balancing performance and computational efficiency in 3D medical image segmentation tasks.