Convolutional, State-Space and Transformer Liver-Tumor Segmentation Models Trained on Hepatocellular Carcinoma Fail...

Architectural innovation is widely assumed to improve the cross-institutional generalization of tumor segmentation models, yet this is rarely tested. We compared three architectural families under an identical, leak-safe protocol in hepatocellular carcinoma (HCC) segmentation.

Convolutional (nnU-Net ResEnc-M), state-space (U-Mamba Bot) and transformer (SwinUNETR) models were trained on HCC-TACE-Seg (n = 103) with identical patient-disjoint five-fold splits, then applied without adaptation to 3D-IRCADb (n = 15) and LiTS (n = 118). We assessed discrimination, case-level failure concordance, lesion-volume dependence, calibration and selective prediction.

Internal performance differed (Dice 0.685, 0.662, 0.479; Friedman p = 0.022) but external performance was low and largely convergent: the convolutional and state-space models were formally equivalent on both external cohorts (pre-specified margin ±0.05 Dice), whereas the transformer was inferior on LiTS (0.222; p < 0.0001) and inconclusive on 3D-IRCADb. Case-level Dice was highly concordant across all pairings (Spearman ρ = 0.84-0.96; 0.67-0.81 on LiTS after adjustment for lesion volume), the same cases failing across architectures. Lesion volume predicted performance in all six architecture × cohort combinations; lesions < 5 mL failed completely in 61-76% of cases and formed 1.9% of the development cohort against 31.4% of LiTS. Neither liver-mask constraint nor ensembling recovered performance. Deferring the 30% least-confident LiTS cases raised Dice to 0.516 (0.450 when only empty predictions were removed).

Under a single-source, TACE-treated HCC training regime, cross-dataset failure of liver-tumor segmentation was governed primarily by lesion size rather than by architectural family; confidence-based selective prediction offers a deployable mitigation.