ENUMERATION OF LONELY PAIRS OF GENE TREES AND SPECIES TREES BY MEANS OF ANTIPODAL CHERRIES

In mathematical phylogenetics, given a rooted binary leaf-labeled gene tree topology G and a rooted binary leaf-labeled species tree topology S with the same leaf labels, a coalescent history represents a possible mapping of the list of gene tree coalescences to the associated branches of the species tree on which those coalescences take place. For certain families of ordered pairs (G, S), the number of coalescent histories increases exponentially or even faster than exponentially with the number of leaves n. Other pairs have only a single coalescent history. We term a pair (G, S) lonely if it has only one coalescent history. Here, we characterize the set of all lonely pairs (G, S). Further, we characterize the set of pairs of rooted binary unlabeled tree shapes at least one of the labelings of which is lonely. We provide formulas for counting lonely pairs and pairs of unlabeled tree shapes with at least one lonely labeling. The lonely pairs provide a set of examples of pairs (G, S) for which the number of compact coalescent histories-which condense coalescent histories into a set of equivalence classes-is equal to the number of coalescent histories. Application of the condition that characterizes lonely pairs can also be used to reduce computation time for the enumeration of coalescent histories.