Mathematical and Simulation-Based Analysis of the Behavior of Admixed Taxa in the Neighbor-Joining Algorithm

Area of Science:

• Phylogenetics
• Computational Biology
• Evolutionary Biology

Background:

• The neighbor-joining (NJ) algorithm is widely used for phylogenetic inference.
• NJ is expected to exhibit specific properties (antecedence of clustering, intermediacy of distances, intermediacy of path lengths) when analyzing distance matrices with admixed taxa.

Purpose of the Study:

• To investigate the behavior of the neighbor-joining algorithm on distance matrices containing admixed taxa.
• To examine the occurrence of three specific NJ properties: antecedence of clustering, intermediacy of distances, and intermediacy of path lengths.

Main Methods:

• Mathematical prediction of property satisfaction frequency in random trees without admixture.
• Simulation of three admixture scenarios: random distance matrices, treelike evolution with admixture, and perturbed treelike scenarios.
• Analysis of distance matrices containing taxa constructed by linear admixture of source taxa distances.

Main Results:

• The three neighbor-joining properties are sometimes violated in distance matrices with admixed taxa, contrary to previous conjectures.
• These properties are satisfied more frequently than expected by chance when an admixed taxon is present.
• Property satisfaction is highest when non-admixed taxa exhibit treelike evolution.

Conclusions:

• The neighbor-joining algorithm's behavior with admixed taxa is complex and does not always conform to expected properties.
• Admixture can lead to violations of specific NJ tree inference properties.
• The findings enhance understanding of evolutionary tree construction from non-treelike evolutionary processes.