[A comparative study of four deep learning instance segmentation models for tooth segmentation and tooth numbering...

Objective: To compare the performance of four deep learning instance segmentation models in tooth contour segmentation and Fédération Dentaire Internationale (FDI) tooth numbering recognition on intraoral occlusal images, and to preliminarily evaluate the clinical acceptability of the best-performing model. Methods: A total of 700 adult participants, comprising 343 males and 357 females aged 35-87 years, including individuals with complete dentition and those with dentition defects, who attended the Department of Prosthodontics, Hospital of Stomatology, China Medical University, between March 2022 and October 2024 and underwent standardized intraoral occlusal photography were enrolled, yielding 1 400 images; after quality screening, 1 300 images from 650 participants were finally included. The dataset was divided by images into a training set (910 images), a validation set (195 images), and a test set (195 images) at a ratio of 7.0∶1.5∶1.5, and only the training set was augmented to 2 730 images. Twenty-eight tooth categories, excluding third molars, were annotated according to the FDI two-digit notation system. Under unified training conditions, four models: Mask R-CNN, RTMDet, YOLOv8-seg, and YOLOv11-seg were compared, and test-set performance was evaluated using segmentation mean average precision at an IoU threshold of 0.5 (seg_mAP@50), precision, recall, and F1-score, while five attending prosthodontists were recruited to compare tooth numbering accuracy with that of the best-performing model and to assess its practical applicability. Results: YOLOv11-seg outperformed the other models across all metrics, achieving a seg_mAP@50 of 98.2%, a precision of 96.3%, a recall of 96.8%, and an F1-score of 96.5%. The tooth numbering accuracy of this model was 97.5% (2 145/2 200), showing no statistically significant difference from the mean accuracy of the five attending prosthodontists (98.6%±1.3%) (t=1.89, P=0.130), and 98.6% of the model outputs were rated as clinically acceptable. Conclusions: YOLOv11-seg achieved highly accurate tooth segmentation and tooth numbering recognition on intraoral occlusal images, with performance comparable to that of attending prosthodontists, thus providing a technical basis for subsequent dentition-defect analysis based on tooth-level information.