Detection of genetic heterogeneity for complex quantitative phenotypes

Statistically characterizing factors responsible for quantitative phenotype expression (e.g., polygenes, major genes, shared household factors, etc.) through model selection strategies is a difficult task. A great deal of effort has been expended on refining mathematical and computational aspects of various segregation models used to characterize unique expressions of quantitative phenotypes in an effort to make these models easier to implement and evaluate for a given set of data. In this paper a slightly different angle is emphasized: namely, the explicit modeling of the potentially numerous heterogeneous genetic and environmental processes (i.e., segregation patterns, household aggregations, etiologic processes, etc.) that could contribute to the overall variation of a quantitative trait. As such, this paper describes tools for detecting quantitative trait heterogeneity that are meant to answer such questions as "are there pedigrees among a great many that show a pattern consistent with a possibly very specific single locus segregation pattern while the rest show compatibility with a polygenic or purely environmental pattern?" Methods for determining the significance of such heterogeneity are also discussed, as are the results of numerous examples and simulation studies carried out in an effort to validate and further elaborate aspects of the proposed techniques.