Flexible estimation of covariance function by penalized spline with application to longitudinal family data

Longitudinal data are routinely collected in biomedical research studies. A natural model describing longitudinal data decomposes an individual's outcome as the sum of a population mean function and random subject-specific deviations. When parametric assumptions are too restrictive, methods modeling the population mean function and the random subject-specific functions nonparametrically are in demand. In some applications, it is desirable to estimate a covariance function of random subject-specific deviations. In this work, flexible yet computationally efficient methods are developed for a general class of semiparametric mixed effects models, where the functional forms of the population mean and the subject-specific curves are unspecified. We estimate nonparametric components of the model by penalized spline (P-spline, Biometrics 2001; 57:253-259), and reparameterize the random curve covariance function by a modified Cholesky decomposition (Biometrics 2002; 58:121-128) which allows for unconstrained estimation of a positive-semidefinite matrix. To provide smooth estimates, we penalize roughness of fitted curves and derive closed-form solutions in the maximization step of an EM algorithm. In addition, we present models and methods for longitudinal family data where subjects in a family are correlated and we decompose the covariance function into a subject-level source and observation-level source. We apply these methods to the multi-level Framingham Heart Study data to estimate age-specific heritability of systolic blood pressure nonparametrically.