Incomplete lineage sorting shaped mixed traits during a colobine primate radiation

Lineages undergoing rapid evolutionary diversification may evolve a set of mixed traits as a result of hybrid speciation, gene flow, or incomplete lineage sorting (ILS). However, how best to quantify these alternative processes impacting phenotypic variation remains unclear. Here, we examined trait evolution in two sister genera of endangered Asian primates, Trachypithecus and Semnopithecus. By assembling a de novo genome of the Trachypithecus pileatus group, which is geographically located in a transitional zone between these two genera, our integrated phylogenomic analyses clarified the evolutionary relationship of the T. pileatus group with other Trachypithecus species. We also identified that morphological similarities shared between this species group and Semnopithecus are the result of ILS, rather than gene flow or hybrid speciation. Across all chromosomes of the T. pileatus group, ILS contributed 8.9% of whole genome segments. Across these segments, we distinguished 77 genes such as FGFBP1 and FOXO1 that are involved in pathways of bone development and osteoblast differentiation. Functional experiments indicate that FGFBP1 genotypes shared by species of the T. pileatus group and Semnopithecus appear to enhance osteogenic capability and mineralization, possibly resulting in larger body size and similarities in skull morphology compared with other species of Trachypithecus. The study reveals that ILS has shaped the evolution of mixed traits within gene-regulatory networks, driving phenotypic diversity during periods of rapid species divergence in this highly successful primate radiation. ILS appears to be more common than previously thought and represents a critical step in accurately assessing the phylogeny of closely related taxa.