Estimating heterogeneity in random effects models for longitudinal data

In this paper, we are interested in estimating parameters entering nonlinear mixed effects models using a likelihood maximization approach. As the accuracy of the likelihood approximation is likely to govern the quality of the derived estimates of both the distribution of the random effects and the fixed parameters, we propose a methodological approach based on the adaptive Gauss Hermite quadrature to better approximate the likelihood function. This work presents improvements of this quadrature that render it accurate and computationally efficient in the problem of likelihood approximation with, an application to mixture models, models which allow the description of coexistence of several different homogeneous subpopulations specifying the distribution of random effects as a mixture of Gaussian distributions. These improvements are based on a new choice of the scaling matrix followed by its optimisation. An application to a phase III clinical trial of an anticoagulant molecule is proposed and estimation results are compared to those obtained with the most frequently used method in population pharmacokinetic analysis. Moreover, in order to evaluate the accuracy of the estimations, an analysis of simulated pharmacokinetic data derived from the model and the a priori values of population parameters of the previous study are presented.