On the genealogy of a rare allele

The gene genealogy is derived for a rare allele that is descended from a mutant ancestor that arose at a fixed time in the past. Following Thompson (1976, Amer. J. Human Genet. 28, 442-452), the fractional linear branching process is used as a model of the demography of a rare allele. The model does not require the total population size to be constant or the mutant class to be neutral; so long as individuals in the class are selectively equivalent, the class as a whole may have a selective advantage, or disadvantage, relative to other alleles in the population. An exact result is given for the joint probability distribution of the coalescence times among a sample of alleles descended from the mutant. A method is described for rapidly simulating these coalescence times. The relationship between the genealogical structure of a discrete generation branching process and a continuous generation birth-death process is elucidated. The theory may be applied to the problem of estimating the ages of rare nonrecurrent mutations.