Coordinate attention and deformable convolution-enhanced YOLO for bilateral maxillary sinus segmentation, a paranasal...

Accurate delineation of the maxillary sinuses in CT images is a relevant step supporting preoperative planning and the evaluation of sinonasal disease within otolaryngology. Manual contouring of these paranasal sinus structures is difficult and error-prone, since the surrounding anatomy is complex, bony structures are thin, and inter-patient variability is large. This paper proposes an instance segmentation framework that extends the YOLO11n-Seg architecture through the integration of Coordinate Attention (CoordAtt) modules and Deformable Convolutional Networks v2 (DCNv2), to improve spatial feature representation and boundary precision for the right and left maxillary sinuses in CT images. CoordAtt encodes directional spatial context along horizontal and vertical axes to enhance anatomical localisation, while DCNv2 replaces selected fixed-kernel convolution layers with adaptive deformable sampling to better capture irregular and thin-walled sinus boundaries. The proposed model was trained and evaluated on the publicly available NasalSeg dataset, using all axial CT slices containing either maxillary sinus from 130 patients (3,620 slices; 104/13/13 patients across training/validation/test), and compared against three YOLO-family baselines (YOLOv8n-Seg, YOLO11n-Seg, and YOLO26n-Seg). On the 382-slice test set (13 patients), the proposed model achieved a mask mAP50 of 93.07% and mask mAP50-95 of 74.49%, outperforming the three baselines by up to +4.60 and +3.30 percentage points, respectively, and a box mAP50 of 91.00% and mAP50-95 of 75.83%. A component-wise ablation study confirmed that this improvement arises from the joint use of CoordAtt and DCNv2 rather than from either mechanism alone, and patient-level bootstrap confidence intervals and paired significance testing confirmed the statistical robustness of the observed gains. These results represent proof-of-concept evidence that coordinate-aware attention and deformable convolution can be effectively integrated into a YOLO-based instance segmentation pipeline for bilateral maxillary sinus delineation from CT.