Related Experiment Video
Updated: Jan 16, 2026
Author Spotlight: 3D Movement Assessment of Maxillary Posterior Teeth in Clear Aligner Treatment
Published on: February 23, 2024
Expert consensus-derived evaluation criteria for orthodontic treatment outcomes using a novel ranking method: A
Huanhuan Chen1, Hanwei Zheng1, Yue Lai1
1Department of Orthodontics, Cranial-Facial Growth and Development Center, National Center of Stomatology, National Clinical Research Center for Oral Diseases, National Engineering Laboratory for Digital and Material Technology of Stomatology, Beijing Key Laboratory for Digital Stomatology, Research Center of Engineering and Technology for Computerized Dentistry Ministry of Health, Peking University School and Hospital of Stomatology, 22, Zhongguancun South Avenue, Haidian District, Beijing 100081, China.
Objective:
This retrospective expert consensus study (PKUSSIRB No.202058145) aimed to establish expert consensus-derived evaluation criteria for orthodontic treatment outcomes using the Merge Ranking Method on post-treatment dental casts.
Material And Methods:
From patients treated at the Department of Orthodontics from January 2018 to December 2022, 216 cases were randomly selected for evaluation by 65 orthodontic experts using the Merge Ranking Method. Concurrently, nine objective indicators of the 216 post-treatment dental casts were measured by three researchers. The consistency analysis of experts' subjective evaluation and researchers' objective measurement was conducted, respectively. Through subjective-to-objective correlation analysis and regression analysis, the objective indicators significantly correlated with experts' subjective evaluations were selected, their weights were determined, and the threshold values of grading evaluation were screened.
Results:
The 65 orthodontic experts demonstrated: (1) moderate pairwise consistency (mean Spearman's ρ=0.560, 95% bootstrap CI: 0.556-0.564), (2) significant group-level concordance across two independent panels (Kendall's W=0.544-0.606, all P<0.001), and (3) near-perfect cross-panel reliability for 24 overlapping cases (Kendall's τ-b=0.833-0.880, P<0.001), confirming panel homogeneity for subsequent analyses. Inter-rater reliability among the three researchers showed excellent consistency (mean ICC=0.835, 95% CI: 0.788-0.882, range: 0.736-0.920), paralleled by high intra-rater reliability (mean ICC=0.832, 95% CI: 0.806-0.858, range: 0.715-0.948) across all 216 cases. Six objective indicators (occlusal relationship, overbite, alignment, overjet, occlusal contact, and buccal-lingual inclination) significantly predicted expert evaluations in a regression model (cumulative R2=0.598, P<0.001). The threshold values for grading orthodontic treatment outcomes as Excellent, Good, Fair, Poor, and Worst were screened to be 1.846, 2.454, 3.492, and 4.312, respectively.
Conclusions:
This expert consensus study demonstrated moderate consistency in subjective orthodontic outcome evaluation, with the occlusal relationship emerging as the primary quality determinant. The developed Merge Ranking Method addressed conventional ranking limitations through its innovative two-stage approach: initial segmented evaluation reduced expert fatigue, while subsequent dynamic adjustments improved borderline case classification.
