Commingling and segregation analyses: comparison of results from a simulation study of a quantitative trait

Commingling analysis is commonly used to provide preliminary evidence for a single genetic locus with a major effect on the quantitative trait of interest. In this paper, the effectiveness of commingling analysis as a screening technique to identify samples for segregation analysis is assessed by applying both commingling and segregation analyses to samples of simulated pedigree data in which a major locus is segregating in the presence of polygenes and an individual-specific environmental effect. Under the circumstances simulated here, there is evidence for a single locus from segregation analysis but not from commingling analysis in at least 20% of the samples. No more than 2% of the samples provided evidence for commingling but not for segregation of a single locus. Comparisons of the samples that give evidence for both commingling and segregation, evidence for one but not the other, and no evidence for either show that evidence for commingling depends on the distributional characteristics of the trait in the sample, while support for the single locus from segregation analysis depends on both the distributional characteristics as well as the transmission of the rarer allele from parents to offspring. Since lack of commingling does not rule out the existence of a single locus in the realistic situations considered here, commingling analysis has limited usefulness as a screening technique for the presence for a single locus. In contrast, evidence for commingling does suggest the possibility that a single locus has a major effect on the trait and commingling analysis can provide guidance in the choice of initial parameter estimates for segregation analysis.