Testing Random Effects in Nonlinear Mixed-Effects Models

Nonlinear mixed-effects modexls are frequently used to analyse longitudinal and clustered data from medical studies with subject-specific variability. A key question in such mixed-effects models is which random effects are truly needed in the model, which amounts to testing whether associated variance components are non-zero. Unlike linear mixed-effects models, testing random effects in nonlinear mixed-effects models is an understudied problem due to their model complexity and convergence issues in practice. Since the null hypothesis lies on the boundary of parameter space, the usual asymptotic chi-squared distribution of likelihood ratio and score tests is incorrect. The correct asymptotic distribution is a mixture of chi-squared distributions, however determining the mixing weights is generally not possible, especially when testing multiple correlated random effects. To address these issues, we propose a flexible nonparametric framework for testing random effects in nonlinear mixed-effects models with additive random errors that does not require normality or any other distribution for random effects and errors. We introduce a flexible test based on a suitable permutation procedure to approximate the finite-sample distribution of our test statistic, which also enjoys distribution-free estimates of variance components. The framework allows users to select among estimation methods based on their data characteristics. Our proposal can be used to test all random effects or any subset of them. We evaluate the performance of our nonparametric method through extensive simulations and two motivating case studies. We provide an R package, called TestREnlme, for the implementation of our proposed tests.