Improving the calculation of statistical significance in genome-wide scans

Calculations of the significance of results from linkage analysis can be performed by simulation or by theoretical approximation, with or without the assumption of perfect marker information. Here we concentrate on theoretical approximation. Our starting point is the asymptotic approximation formula presented by Lander and Kruglyak (1995, Nature Genetics, 11, 241--247), incorporating the effect of finite marker spacing as suggested by Feingold et al. (1993, American Journal of Human Genetics, 53, 234--251). We consider two distinct ways in which this formula can be improved. Firstly, we present a formula for calculating the crossover rate rho for a pedigree of general structure. For a pedigree set, these values may then be weighted into an overall crossover rate which can be used as input to the original approximation formula. Secondly, the unadjusted p -value formula is based on the assumption of a Normally distributed nonparametric linkage (NPL) score. This leads to conservative or anti-conservative p -values of varying magnitude depending on the pedigree set structure. We adjust for non-Normality by calculating the marginal distribution of the NPL score under the null hypothesis of no linkage with an arbitrarily small error. The NPL score is then transformed to have a marginal standard Normal distribution and the transformed maximal NPL score, together with a slightly corrected value of the overall crossover rate, is inserted into the original formula in order to calculate the p -value. We use pedigrees of seven different structures to compare the performance of our suggested approximation formula to the original approximation formula, with and without skewness correction, and to results found by simulation. We also apply the suggested formula to two real pedigree set structure examples. Our method generally seems to provide improved behavior, especially for pedigree sets which show clear departure from Normality, in relation to the competing approximations.