[Interpretable multimodal deep learning for orthodontic extraction planning in class Ⅱ malocclusion]

Objective: To develop an interpretable multimodal deep learning-based two-stage task-specific prediction framework for orthodontic extraction need and extraction pattern prediction in patients with class Ⅱ malocclusion, so as to provid auxiliary reference for clinical orthodontic extraction decision-making. Methods: This retrospective study collected data on patients with class Ⅱ malocclusion who visited the Department of Orthodontics, School of Stomatology, The Fourth Military Medical University, from January 2014 to December 2023. An extraction need prediction dataset (n=824) and an extraction pattern prediction dataset (n=610) were established, with a three-level annotation procedure used to label the need for extraction and specific extraction patterns. The You Only Look Once version 5 (YOLOv5) model was used to extract regions of interest (ROI) from lateral cephalometric radiographs and extraoral lateral photographs. For lateral cephalometric radiographs, the cropping region included key cephalometric landmarks, with boundaries encompassing the nasion (N), menton (Me), basion (Ba), and pronasale (Prn). For extraoral lateral photographs, the cropping region encompassed the nasal tip, the most prominent points of the upper and lower lips, and the lowest point of the soft-tissue chin. A two-stage task-specific prediction framework was developed. An extraction need prediction model was developed using dual-view inputs of intraoral lateral photographs and extraoral lateral photographs. A separate extraction pattern prediction model, based on multimodal multi-view inputs (intraoral lateral photographs, extraoral lateral photographs, and lateral cephalometric radiographs) was trained using the AlexNet architecture. Model performance was evaluated using accuracy, precision, recall, F1-score, and the area under the receiver operating characteristic curve (AUC). Gradient-weighted class activation mapping (Grad-CAM) was used to visualize and analyze the basis of model decisions. Results: For extraction need prediction, the AlexNet model showed relatively favorable overall performance. The accuracies of the intraoral lateral photograph single-view, extraoral lateral photograph single-view, and intraoral-extraoral dual-view clinical photograph datasets were 84.3%, 85.8%, and 82.0%, respectively, with corresponding AUC of 0.92, 0.90, and 0.84. In extraction pattern prediction, the multimodal multi-view AlexNet model integrating three types of images achieved an accuracy of 88.0% and an AUC of 0.92. For the two extraction patterns, namely extraction of four first premolars and extraction of two maxillary first premolars plus two mandibular second premolars, the macro-average precision, recall, and F1-score were 87.4%, 86.9%, and 87.1%, respectively. Grad-CAM visualization showed that the high-weight regions involved in model decision-making were generally consistent with the key anatomical regions considered in clinical extraction decision-making. Conclusions: The two-stage deep learning model developed in this study effectively predicted orthodontic extraction need and extraction patterns in patients with class Ⅱ malocclusion on the present dataset. Grad-CAM visualization suggested that the model mainly focused on anatomical regions relevant to clinical extraction decision-making and could provide auxiliary reference for clinical decision-making.