F_ST and the triangle inequality for biallelic markers

Area of Science:

• Population Genetics
• Evolutionary Biology
• Bioinformatics

Background:

• The population differentiation statistic F_ST is widely used as a pairwise distance measure between populations.
• Sewall Wright recognized that F_ST is not a true metric due to violations of the triangle inequality under certain allele frequency conditions.

Purpose of the Study:

• To rigorously prove that F_ST *never* satisfies the triangle inequality for biallelic markers with differing allele frequencies across three populations.
• To quantify the deviation from the triangle inequality and identify conditions for maximal deviation.
• To explore the implications of these F_ST properties for phylogenetic analyses and population genetic data interpretation.

Main Methods:

• Theoretical analysis of F_ST properties concerning the triangle inequality.
• Mathematical modeling to determine allele frequency vectors causing maximal deviation.
• Empirical evaluation using human genomic data to assess real-world F_ST deviations.
• Examination of consequences for phylogenetic methods (e.g., neighbor-joining trees) and visualization techniques (e.g., multidimensional scaling).

Main Results:

• F_ST fundamentally fails to satisfy the triangle inequality for any three populations with differing allele frequencies at biallelic markers.
• The deviation from the triangle inequality is maximized under specific allele frequency configurations.
• Analysis of human genomic data reveals loci exhibiting F_ST deviations close to the theoretical maximum.
• Failure of the triangle inequality has significant consequences for the accuracy of evolutionary tree reconstructions and distance-based analyses.

Conclusions:

• F_ST is not a reliable metric for population distances, necessitating caution in its interpretation.
• The theoretical findings provide a framework for understanding F_ST's limitations in population structure analysis.
• Researchers should be aware of these limitations when employing F_ST in phylogenetic inference and multidimensional scaling.