Prediction discrepancies for the evaluation of nonlinear mixed-effects models

Area of Science:

• Statistics
• Pharmacometrics
• Computational Biology

Background:

• Nonlinear mixed-effects (NLME) models are increasingly utilized in various scientific fields.
• However, limited statistical methods exist for rigorously evaluating these complex models.
• Existing evaluation techniques often rely on approximations, potentially limiting their accuracy.

Purpose of the Study:

• To develop and validate a novel criterion and test for evaluating NLME models.
• To assess model performance based on the entire predictive distribution, not just point estimates.
• To compare the proposed method with existing evaluation techniques like standardized prediction errors (SPE).

Main Methods:

• Development of a prediction discrepancy (pd) metric, representing the percentile of an observation within the marginal predictive distribution.
• Utilizing Monte Carlo integration for computing pd, avoiding model approximations.
• Application of the Kolmogorov-Smirnov test to assess the uniform distribution of pd under the null hypothesis.
• Comparison with standardized prediction errors (SPE) using a simulation study on a pharmacokinetic model.

Main Results:

• The proposed prediction discrepancy method provides a comprehensive evaluation of NLME models.
• Prediction discrepancies (pd) are shown to be uniformly distributed between 0 and 1 for valid models.
• The new criterion demonstrates advantages over standardized prediction errors (SPE), especially when model approximations are used.
• Graphical analysis of pd trends can reveal specific model misspecifications.

Conclusions:

• The developed prediction discrepancy criterion and test offer a reliable and model-approximation-free method for evaluating NLME models.
• This approach enhances the assessment of model fit and predictive performance.
• The method provides valuable insights into model validity and potential areas for improvement.