Genealogy of neutral genes in two partially isolated populations

Gene genealogy in two partially isolated populations which diverged at a given time t in the past and have since been exchanging individuals at a constant rate m is studied based upon an analytic method for large t and a simulation method for any t. Particular attention is paid to the conditions under which neutral genes sampled from populations are mono-, para-, and polyphyletic in terms of coalescence (divergence) times of genes. It is shown tha the probability of monophyly is high if M = 2Nm less than 0.5 and T = t/(2N) greater than 1, where N is the size of ancestral and descendant haploid populations, in which case most gene genealogies are likely to be concordant with the population relatedness. This probbility decreases as the sample size of genes increases. On the other hand, the case where the probability of monophyly is low will be either that of M greater than 1 and any T or that of M less than 1 and T less than 1, but the clear distinction between these conditions appears very difficult to make. These results are also examined if the gene genealogy is reconstructed from nucleotide differences. It is then shown that the results based upon coalescence times remain valid if the number of nucleotide differences between any pair of genes is not much smaller than 10. To observe such large nucleotide differences in small populations and therefore infer a reliable gene genealogy, we must examine a fairly long stretch of DNA sequences.