Deep learning for automatic segmentation of the inferior alveolar nerve using a hybrid CNN-transformer framework

Accurate identification of the inferior alveolar nerve (IAN) is essential for preventing nerve injury during dental and maxillofacial procedures such as tooth extraction, implant placement, and orthognathic surgery. However, manual annotation of the IAN in cone-beam computed tomography (CBCT) is challenging because of image noise, anatomical variability, and the complex three-dimensional curvature of the nerve pathway. In this study, we propose an improved automatic segmentation framework for IAN identification in CBCTs using a hybrid CNN-attention architecture built upon the nnU-Net framework. The proposed framework introduces a task-specific CNN-attention hybrid architecture with stage-restricted Permuted Adaptive Instance Normalization (Permuted AdaIN) and decoder-stage contextual refinement to improve robustness to appearance variability while preserving anatomical continuity in thin tubular nerve segmentation. Permuted AdaIN is selectively applied to encoder stage 2 to encourage appearance-invariant feature representations while preserving boundary-sensitive anatomical structures. In addition, the decoder attention module refines contextual interactions between encoder and decoder features after skip fusion, improving segmentation consistency of the thin tubular nerve structure. A total of 130 CBCTs from two institutions were used for training and evaluation. The proposed method achieved Dice similarity coefficients of 0.63 ± 0.17 on the internal dataset (KUAH) and 0.62 ± 0.12 on the external dataset (KUGH), showing a modest numerical improvement compared with nnU-Net (0.60 ± 0.18 and 0.59 ± 0.11, respectively). In addition, the proposed method achieved lower boundary errors, with HD95 values of 2.96 ± 1.27 mm and 3.72 ± 7.63 mm, and ASSD values of 1.22 ± 0.49 mm and 1.29 ± 2.10 mm for the internal and external datasets, respectively. Qualitative analysis further demonstrated improved boundary alignment and continuity of the segmented nerve trajectory. These results suggest that the proposed framework may improve segmentation consistency and boundary agreement in automated IAN segmentation on CBCT images, although the magnitude of improvement over nnU-Net should be interpreted cautiously.