Segmenting scaffolds with ingrown bone in micro-CT images: considerations of evaluation metrics and loss functions

U-Net models are useful for biomedical image segmentation because they do not require large training sets. Here, we trained three U-Net models, each with a different loss function, to segment micro-CT image patches of scaffolds with ingrown bone to quantify bone growth fraction. Segmentation performance is typically reported using metrics like Accuracy, DSC, Precision, and Recall. However, these metrics did not reflect performance relative to the application's key output measure. We therefore aimed to establish an application-specific parameter, ΔBGF: the difference in bone growth fraction (BGF) between predicted output (PO) and ground truth (GT). Bland-Altman analysis showed small systematic bias (< 1.5 percentage points), supporting PO-GT agreement. Accuracy was the metric most strongly correlated to |ΔBGF| in nearly all comparisons (Steiger test), and this relationship was stable under bootstrap resampling. The models were robust to changes in orientation, resolution, and contrast, supporting broader generalizability. The cross-entropy loss model generated the most reliable segmentation. PO-GT error was small relative to meaningful biological differences, suggesting it is unlikely to affect future treatment comparisons. Overall, standard metrics can be misleading, while ΔBGF provides a reliable, application-specific measure of performance for this still developing use of U-Net in quantifying bone regeneration.