Deep Learning-Based Multi-Stage System for Automated Tooth Detection and Segmentation in Orthodontic Photography

This study aimed to develop and evaluate a fully automated, artificial intelligence-driven system for tooth detection and segmentation from complete orthodontic photographic plates, with the goal of improving clinical outcomes and workflow efficiency in dentistry. Manual tooth charting and segmentation are often labor-intensive, prone to human error, and inconsistent across clinicians, potentially compromising diagnosis, treatment planning, and long-term patient records. A dataset of 2000 composite dental images was annotated and divided into training (n = 1600), validation (n = 200), and testing (n = 200) sets. Model performance was further assessed on an independent external validation dataset obtained from a separate dental institution. The proposed multi-stage system consisted of intraoral view identification and maxilla/mandible localization using YOLOv11 (referred to as YOLO-Arch), followed by tooth-level detection with a separate YOLOv11 model (referred to as YOLO-Tooth), and finally pixel-level segmentation using the Segment Anything Model (SAM). Each detected tooth was assigned a class label and confidence score, and segmentation masks were generated in a zero-shot manner without additional training. The system achieved excellent performance, with 99.5% mAP@0.5 and 96.2% mAP@0.5-0.95 for arch classification, along with near-perfect precision (0.999) and recall (1.0). For tooth detection, the model attained an mAP@0.5 of 0.943 for the maxillary arch and 0.918 for the mandibular arch, corresponding to mean precision/recall values of 0.906/0.905 and 0.875/0.871, respectively. Using the predefined quantitative criteria for clinical usability, 94% of the generated segmentation masks met all requirements for use without correction. The remaining 6% primarily failed due to localized boundary inaccuracies at interproximal regions or minor anatomical inconsistencies, rather than gross segmentation errors. The complete inference pipeline processed each image in an average of 2.2 s, demonstrating practical applicability in clinical workflows. Overall, the integration of YOLO and SAM provides a fast, accurate, and fully automated solution for tooth analysis in orthodontic photography, supporting large-scale annotation, digital charting, and AI-assisted diagnostics in both orthodontic and general dental practice.