The genetic structure of admixed populations

A method for simultaneously estimating the admixture proportions of a hybrid population and Wright's fixation index, FST, for that hybrid is presented. It is shown that the variance of admixture estimates can be partitioned into two components: (1) due to sample size, and (2) due to evolutionary variance (i.e., genetic drift). A chi-square test used to detect heterogeneity of admixture estimates from different alleles, or loci, can now be corrected for both sources of random errors. Hence, its value for the detection of natural selection from heterogeneous admixture estimates is improved. The estimation and testing procedures described above are independent of the dynamics of the admixture process. However, when the admixture dynamics can be specified, FST can be predicted from genetic principles. Two admixture models are considered here, gene flow and intermixture. These models are of value because they lead to very different predictions regarding the accumulation of genes from the parental populations and the accumulation of variance due to genetic drift. When there is not evidence for natural selection, and it is appropriate to apply these models to data, the variance effective size (Ne) of the hybrid population can be estimated. Applications are made to three human populations: two of these are Afro-American populations and one is a Yanomamö Indian village. Natural selection could not be detected using the chi-square test in any of these populations. However, estimates of effective population sizes do lead to a richer description of the genetic structure of these populations.