Asymptotic distribution of the "orthogonal" quantitative transmission disequilibrium test in a structured population:...

Population structure is a recurrent problem for the detection of associations between a marker and a trait, because it can lead to an excess of false positives of the association tests. One popular way of circumventing this problem is the use of family based tests, which consider the transmission of the genotype from the parents to the offspring. Here we focus on quantitative traits and study the Abecasis "orthogonal" quantitative transmission disequilibrium test, which is commonly used in family based association studies. We derive the probability distribution of this test under a general model of structured population. Our derivations show that this test leads to a small excess of false positives due to population structure. They also illustrate and quantify how the heterogeneity in genotypes and phenotypes between populations affect the power of the test. We finally show that the excess of false positives observed for the Abecasis "orthogonal" test may also be found for the Allison "linear" test, though at a lower extent.