Nonlinear mixed-effects models with misspecified random-effects distribution

Nonlinear mixed-effects models are being widely used for the analysis of longitudinal data, especially from pharmaceutical research. They use random effects which are latent and unobservable variables so the random-effects distribution is subject to misspecification in practice. In this paper, we first study the consequences of misspecifying the random-effects distribution in nonlinear mixed-effects models. Our study is focused on Gauss-Hermite quadrature, which is now the routine method for calculation of the marginal likelihood in mixed models. We then present a formal diagnostic test to check the appropriateness of the assumed random-effects distribution in nonlinear mixed-effects models, which is very useful for real data analysis. Our findings show that the estimates of fixed-effects parameters in nonlinear mixed-effects models are generally robust to deviations from normality of the random-effects distribution, but the estimates of variance components are very sensitive to the distributional assumption of random effects. Furthermore, a misspecified random-effects distribution will either overestimate or underestimate the predictions of random effects. We illustrate the results using a real data application from an intensive pharmacokinetic study.