Evaluating Modeling and Validation Strategies for Tooth Loss.

Area of Science:

• Dental research
• Periodontology
• Biostatistics
• Machine learning in healthcare

Background:

• Prediction modeling is increasingly utilized in dental research for conditions like periodontitis.
• Evaluating tooth loss prediction models requires careful consideration of development and validation methodologies.
• Tooth loss in periodontitis patients is a critical outcome necessitating accurate predictive tools.

Purpose of the Study:

• To assess the impact of various model development and validation strategies on the predictive performance of tooth loss models in periodontitis patients.
• To compare different machine learning algorithms and data handling techniques for predicting tooth loss.
• To determine the reliability of internal versus external validation for tooth loss prediction models.

Main Methods:

• Two independent cohorts of periodontitis patients were analyzed over extended follow-up periods.
• Tooth loss and patient/tooth-level predictors were recorded, with tooth loss being a relatively rare event.
• Model performance was evaluated across different complexities (logistic regression to extreme gradient boosting), sample sizes, prediction periods, and validation schemes (internal vs. external).

Main Results:

• All models exhibited limited sensitivity but high specificity for predicting tooth loss.
• Internal validation significantly overestimated predictive power (AUC up to 0.90) compared to external validation (AUC 0.62-0.82).
• More complex models did not consistently outperform simpler ones, and reduced sample size negatively impacted performance, especially for complex models.

Conclusions:

• None of the developed tooth loss prediction models demonstrated sufficient clinical utility, despite high in-sample accuracy.
• Rigorous model development and, critically, external validation are essential and must be appropriately reported.
• Temporal validation proved to be a viable option, yielding results comparable to the base case when models were trained and tested across different time periods.