Estimation of genetic effects in multiple cases family studies using penalized maximum likelihood methodology

Family studies are often used in genetic research to explore associations between genetic markers and various phenotypes. A commonly used design oversamples families enriched with the disease under study for efficient data collection and estimation. For instance, in a multiple cases family study, families are selected based on the number of affected relatives. In such cases, valid inference for the model parameters relies on the proper modeling of both the within family correlations and the outcome-dependent sampling, also known as ascertainment. A flexible modeling approach is the ascertainment-corrected mixed-effects model, but it is known to only be asymptotically identifiable, because in small samples the available data do not provide sufficient information to estimate both the intercept and the genetic variance. To deal with this issue, we propose a penalized maximum likelihood estimation procedure which reliably estimates the model parameters in small family studies by using external population-based information.