Challenges in Species Tree Estimation Under the Multispecies Coalescent Model

Area of Science:

• Computational evolutionary biology
• Phylogenetics
• Population genetics

Background:

• The multispecies coalescent (MSC) model is crucial for species phylogeny inference, accounting for ancestral polymorphism and gene tree conflict.
• Existing methods for species tree estimation under MSC fall into two categories: computationally intensive full likelihood methods and faster approximate or summary coalescent methods.

Purpose of the Study:

• To review the statistical nature of species tree estimation under the MSC.
• To explore conceptual issues and challenges by analyzing simple cases of three or four closely related species.
• To compare the statistical performance of full likelihood and summary coalescent methods.

Main Methods:

• Mathematical analysis of species tree estimation under the MSC.
• Computer simulations to evaluate method performance.
• Focus on simple phylogenetic scenarios (3-4 species) to illustrate key differences.

Main Results:

• Significant differences in statistical performance exist between full likelihood and summary coalescent methods.
• Summary methods can exhibit counterintuitive behaviors like parameter unidentifiability and inconsistency in the anomaly zone due to inefficient data utilization.
• These issues observed with summary methods are resolved when using full likelihood methods.

Conclusions:

• Full likelihood methods offer more statistically robust species tree inference under the MSC, despite higher computational demands.
• Improving the computational efficiency and model realism of likelihood methods is essential.
• Enhancing the statistical efficiency of summary methods is also critical for broader applicability.