A Volumetric Anatomy-Aware, Prompt-Guided Adaptation Framework for the Segment Anything Model in Head and Neck...

Background Head and neck (H&N) radiotherapy planning requires accurate segmentation of numerous organs at risk (OARs), yet many critical structures, such as the optic nerves, chiasm, and cochleae, are small, poorly contrasted on CT, and prone to high inter-observer variability in manual contouring. Recent foundation models like Segment Anything Model (SAM) and its 3D medical variants (SAM-Med3D) offer prompt-based, interactive segmentation, but their real-world utility for H&N small OARs is hindered by unreliable prompt sources and unstable performance on fine anatomical targets. To address these issues, this work proposes an anatomy-aware prompt generation and prompt-guided adaptation tailored specifically for small-structure segmentation to improve both precision and automation feasibility. Purpose To address these challenges, we developed a domain-adapted framework, SAM-FT-HN, by fine-tuning a volumetric SAM backbone (SAM-Med3D) for automated delineation of H&N OARs in radiotherapy planning. Our approach incorporates a coarse-to-fine pipeline in which a lightweight localization network automatically generates organ-specific prompts and candidate regions, followed by a fine-tuned volumetric SAM backbone for refinement. We further introduced task-specific strategies including region-of-interest (ROI)-guided tiny-organ enhancement and organ-specific threshold calibration to improve performance on small structures. The primary objective is to develop and evaluate a fully automated, anatomy-aware adaptation of SAM-Med3D for H&N OAR segmentation, with a secondary objective of improving small-organ segmentation. We systematically compared SAM-FT-HN against baseline SAM-Med3D and nnUNet models quantitatively and qualitatively. Methods We used an institutional H&N CT dataset of 211 patients with manual delineations of 23 OARs, standardized for training, validation, and testing. Eighty percent of cases were used for training and validation, and twenty percent for testing. The testing phase used CT volumes only, and ground-truth masks were used solely for metric calculation. A coarse localization network was trained to generate automatic prompts and ROI candidates. SAM-FT-HN was fine-tuned using task-specific adaptations, and small-organ refinement was performed using ROI-guided enhancement and organ-specific threshold calibration. nnUNet was trained using its automated configuration pipeline. Performance was evaluated using overlap and boundary metrics. Results For large OARs, SAM-FT-HN achieved a mean Dice of 0.814 compared with 0.791 for nnUNet and substantially higher than SAM-Med3D (0.609). For small organs, SAM-FT-HN improved the mean Dice from 0.300 to 0.669 compared with SAM-Med3D and reduced HD95 by approximately 65%, although nnUNet remained higher in average small-organ Dice (0.720). Overall, across all organs, SAM-FT-HN achieved a mean Dice score of 0.770, compared with 0.769 for nnUNet. Overall, SAM-FT-HN showed no statistically significant difference from nnUNet in the patient-level analysis, while substantially outperforming the original SAM-Med3D baseline. Conclusion Fine-tuning a foundation model with anatomy-aware automatic prompting, coarse-to-fine localization, and adaptive thresholding markedly improves small-organ segmentation, substantially mitigating a key limitation of the original foundation model. SAM-FT-HN offers a competitive approach for H&N radiotherapy contouring, with potential for future adaptive workflows.