Tropical Density Estimation of Phylogenetic Trees.

Area of Science:

• Phylogenetics and Evolutionary Biology
• Computational Biology
• Tropical Geometry

Background:

• Gene trees, derived from different genes, often exhibit topological incongruence due to evolutionary events like recombination or horizontal gene transfer.
• Despite incongruence, most gene trees are generally constrained by the species tree topology, representing the overarching evolutionary history.
• Identifying 'outlying' gene trees that deviate from the main distribution is crucial for understanding complex evolutionary dynamics.

Purpose of the Study:

• To propose and apply a novel non-parametric method for estimating gene tree distributions using tropical geometry and a tropical metric.
• To develop an analogue of Kernel Density Estimation (KDE) within the framework of tropical geometry.
• To compare the performance of the proposed tropical metric-based KDE with the existing Billera-Holmes-Vogtmann (BHV) metric.

Main Methods:

• Application of the tropical metric, based on max-plus algebra, to measure distances between phylogenetic trees.
• Development of a tropical geometry-based Kernel Density Estimator (KDE) for non-parametric estimation of gene tree distributions.
• Estimation of tree probabilities using empirical frequencies of nearby trees, weighted by the tropical metric.

Main Results:

• The tropical metric-based KDE demonstrated superior performance compared to the BHV metric in terms of computational efficiency and accuracy using simulated data from the multispecies coalescent model.
• The method successfully identified gene tree distributions and was applied to real-world data from Apicomplexa.

Conclusions:

• The tropical metric provides an effective tool for non-parametric estimation of gene tree distributions, offering advantages over existing metrics.
• This approach enhances the ability to detect phylogenetic incongruence and uncover unusual evolutionary events.
• The method shows promise for applications in diverse biological datasets, including Apicomplexa.