Effective number of different populations: A new concept and how to use it

Widely used methods to assess population genetic structure and differentiation rely on independence of marker loci. Following the assumption, the common metrics, for example F _ST, evaluate genetic structure by averaging across loci. Common metrics do not use information in the associations among loci at the individual level and are often criticized for failing to measure true differentiation even when loci segregate independently. We introduce a new concept to measure β-variation (Effective Number of Different Populations, ENDP). It requires the following steps: (a) calculation of a proper dissimilarity between genetic profiles of all individuals; (b) calculation of suitable pairwise distances between the samples based on the dissimilarities between individuals; (c) calculation of diversity (in terms of Hill numbers) and dispersion of samples based on the pairwise distances between samples; (d) ENDP is then estimated by combining the diversity and dispersion. ENDP estimates β-variation independently of estimates of within-sample α-variation, although β- and α-estimates could statistically correlate to some extent. This new concept differs from the existing standard where β-diversity is estimated based on the "partition of variation" scheme ( $$ or $$ ), so that estimates of β-diversity directly depend on the corresponding values of α-diversity. ENDP estimates are obtained by evaluating information in the available genetic profiles of individuals including association of loci. Therefore, ENDP can be used even in an absence of panmixia. We illustrate the use of this concept by analyzing the population genetic structure of a sexual species (a trematode parasite) occupying connected populations across a broad geographic area. The analysis is complicated by geographically coexisting cryptic species and the potential mixed-mating system of this hermaphroditic parasite. Analyses with subsampled data demonstrated that ENDP estimates are robust. Number of loci used for genotyping has much stronger effect on variation of point ENDP estimates than sample size.