Comparison of different genetic distances to test isolation by distance between populations

Studying isolation by distance can provide useful demographic information. To analyze isolation by distance from molecular data, one can use some kind of genetic distance or coalescent simulations. Molecular markers can often display technical caveats, such as PCR-based amplification failures (null alleles, allelic dropouts). These problems can alter population parameter inferences that can be extracted from molecular data. In this simulation study, we analyze the behavior of different genetic distances in Island (null hypothesis) and stepping stone models displaying varying neighborhood sizes. Impact of null alleles of increasing frequency is also studied. In stepping stone models without null alleles, the best statistic to detect isolation by distance in most situations is the chord distance D_CSE. Nevertheless, for markers with genetic diversities H_S<0.4-0.5, all statistics tend to display the same statistical power. Marginal sub-populations behave as smaller neighborhoods. Metapopulations composed of small sub-population numbers thus display smaller neighborhood sizes. When null alleles are introduced, the power of detection of isolation by distance is significantly reduced and D_CSE remains the most powerful genetic distance. We also show that the proportion of null allelic states interact with the slope of the regression of F_ST/(1-F_ST) as a function of geographic distance. This can have important consequences on inferences that can be made from such data. Nevertheless, Chapuis and Estoup's FreeNA correction for null alleles provides very good results in most situations. We finally use our conclusions for reanalyzing and reinterpreting some published data sets.