Likelihood ratio tests in behavioral genetics: problems and solutions

The likelihood ratio test of nested models for family data plays an important role in the assessment of genetic and environmental influences on the variation in traits. The test is routinely based on the assumption that the test statistic follows a chi-square distribution under the null, with the number of restricted parameters as degrees of freedom. However, tests of variance components constrained to be non-negative correspond to tests of parameters on the boundary of the parameter space. In this situation the standard test procedure provides too large p-values and the use of the Akaike Information Criterion (AIC) or the Bayesian Information Criterion (BIC) for model selection is problematic. Focusing on the classical ACE twin model for univariate traits, we adapt existing theory to show that the asymptotic distribution for the likelihood ratio statistic is a mixture of chi-square distributions, and we derive the mixing probabilities. We conclude that when testing the AE or the CE model against the ACE model, the p-values obtained from using the chi(2)(1 df) as the reference distribution should be halved. When the E model is tested against the ACE model, a mixture of chi(2)(0 df), chi(2)(1 df) and chi(2)(2 df) should be used as the reference distribution, and we provide a simple formula to compute the mixing probabilities. Similar results for tests of the AE, DE and E models against the ADE model are also derived. Failing to use the appropriate reference distribution can lead to invalid conclusions.