AMOVA-based clustering of population genetic data

Determining the genetic structure of populations is becoming an increasingly important aspect of genetic studies. One of the most frequently used methods is the calculation of F-statistics using an Analysis of Molecular Variance (AMOVA). However, this has the drawback that the population hierarchy has to be known a priori. Therefore, the population structure is often based on the results of a clustering analysis. Here I show how these two steps, clustering and calculation of F-statistics, can be combined in a single analysis. I do this by showing how the AMOVA framework is theoretically related to the widely used method of K-means clustering and can be used for the clustering of populations into groups. Simulations were used to show that the method performed very well both under random mating and under nonrandom mating. However, when the migration rates were high, the results were better under random mating than under predominant selfing or clonal reproduction. Two summary statistics were tested for estimating the number of clusters. Overall, pseudo-F showed the better performance, but BIC is better for detecting whether any significant structure is present. The results show that the AMOVA-based K-means clustering is useful for clustering population genetic data. Programs to perform the clustering can be downloaded from www.patrickmeirmans.com/software.