Testing quantitative trait locus effects in genetic backcross studies with double recombination occurring

Testing the existence of quantitative trait locus (QTL) effects is an important task in QTL mapping studies. In this paper, we assume the phenotype distributions from a location-scale distribution family, and consider to test the QTL effects in both location and scale in the backcross studies with double recombination occurring. Without equal scale assumption, the log-likelihood function is unbounded, which leads to the traditional likelihood ratio test being invalid. To deal with this problem, we propose a penalized likelihood ratio test (PLRT) for testing the QTL effects. The null limiting distribution of the PLRT is shown to be a supremum of a chi-square process. As a complement, we also investigate the null limiting distribution of the likelihood ratio test for the case with equal scale assumption. The limiting distributions of the two tests under local alternatives are also studied. Simulation studies are performed to evaluate the asymptotic results and a real-data example is given for illustration.