A Beta-mixture model for assessing genetic population structure

An important fraction of recently generated molecular data is dominant markers. They contain substantial information about genetic variation but dominance makes it impossible to apply standard techniques to calculate measures of genetic differentiation, such as F-statistics. In this article, we propose a new Bayesian beta-mixture model that more accurately describes the genetic structure from dominant markers and estimates multiple F(ST) s from the sample. The model also has important application for codominant markers and single-nucleotide polymorphism (SNP) data. The number of F(ST)  is assumed unknown beforehand and follows a random distribution. The reversible jump algorithm is used to estimate the unknown number of multiple F(ST) s. We evaluate the performance of three split proposals and the overall performance of the proposed model based on simulated dominant marker data. The model could reliably identify and estimate a spectrum of degrees of genetic differentiation present in multiple loci. The estimates of F(ST) s also incorporate uncertainty about the magnitude of within-population inbreeding coefficient. We illustrate the method with two examples, one using dominant marker data from a rare orchid and the other using codominant marker data from human populations.