Penalized variance components for association of multiple genes with traits

Variance component models have gained popularity for genetic analyses, driven by their flexibility to simultaneously analyze multiple genetic variants in a gene by kernel statistics, and their ability to account for population stratification via genomic relationship matrices. For exploratory analyses with modest sample sizes and a potentially large number of variance components, it can be challenging to use standard maximum-likelihood or restricted maximum-likelihood methods to estimate variance components, because these iterative methods often fail to converge when likelihood surfaces are fairly flat, and standard-likelihood ratio statistical tests are not adequate. To overcome these limitations, we developed a penalized-likelihood model, whereby the penalty function follows the popular elastic-net approach, applying both L1 and L2 penalties to the variance components. By simulations, we demonstrate the potential gain in power by using both L1 and L2 penalties, and results from our simulations suggest that assigning 80% of the penalty parameter to the L1 penalty and 20% to the L2 penalty provides a reasonable balance between false-positive and false-negative results. Larger sample size improves the properties of our methods, at the cost of longer computation time. Application of our methods to a study of the influence of DNA methylation on levels of cortisol in reaction to stress testing shows how our method can be used to prioritize findings for further functional studies.