Detecting epistasis with restricted response patterns in pairs of biallelic loci

Well-established examples of genetic epistasis between a pair of loci typically show characteristic patterns of phenotypic distributions in joint genotype tables. However, inferring epistasis given such data is difficult due to the lack of power in commonly used approaches, which decompose the epistatic patterns into main plus interaction effects followed by testing the interaction term. Testing additive-only or all terms may have more power, but they are sensitive to nonepistatic patterns. Alternatively, the epistatic patterns of interest can be enumerated and the best matching one is found by searching through the possibilities. Although this approach requires multiple testing correction over possible patterns, each pattern can be fitted with a regression model with just one degree of freedom and thus the overall power can still be high, if the number of possible patterns is limited. Here we compare the power of the linear decomposition and pattern search methods, by applying them to simulated data generated under several patterns of joint genotype effects with simple biological interpretations. Interaction-only tests are the least powerful; while pattern search approach is the most powerful if the range of possibilities is restricted, but still includes the true pattern.