Estimating data transformations in nonlinear mixed effects models

A routine practice in the analysis of repeated measurement data is to represent individual responses by a mixed effects model on some transformed scale. For example, for pharmacokinetic, growth, and other data, both the response and the regression model are typically transformed to achieve approximate within-individual normality and constant variance on the new scale; however, the choice of transformation is often made subjectively or by default, with adoption of a standard choice such as the log. We propose a mixed effects framework based on the transform-both-sides model, where the transformation is represented by a monotone parametric function and is estimated from the data. For this model, we describe a practical fitting strategy based on approximation of the marginal likelihood. Inference is complicated by the fact that estimation of the transformation requires modification of the usual standard errors for estimators of fixed effects; however, we show that, under conditions relevant to common applications, this complication is asymptotically negligible, allowing straightforward implementation via standard software.